You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2022/07/11 16:30:51 UTC
[GitHub] [nifi] kulikg opened a new pull request, #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
kulikg opened a new pull request, #6192:
URL: https://github.com/apache/nifi/pull/6192
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
Added ListSmb processor and SmbConnectionPoolService
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [x] Pull Request based on current revision of the `main` branch
- [x] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [ ] Build completed using `mvn clean install -P contrib-check`
- [ ] JDK 8
- [ ] JDK 11
- [ ] JDK 17
### Licensing
- [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] kulikg commented on pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
kulikg commented on PR #6192:
URL: https://github.com/apache/nifi/pull/6192#issuecomment-1184049956
>
Of course it is possible! In fact I'm already working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r928309177
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-nar/pom.xml:
##########
@@ -35,5 +35,16 @@
<artifactId>nifi-smb-processors</artifactId>
<version>1.17.0-SNAPSHOT</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smbj-client</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </dependency>
Review Comment:
@kulikg Please delete this dependency. Otherwise the controller service is duplicated on the UI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on PR #6192:
URL: https://github.com/apache/nifi/pull/6192#issuecomment-1185675042
> @kulikg - Did you have the intention of refactoring the existing GetSmb and PutSmb processors to utilise the client? Or do you think this is a body of work for later?
As @kulikg mentioned, updating the existing processors to leverage the Controller Service could be something to consider in a separate issue after this one is completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] lirontb commented on pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
lirontb commented on PR #6192:
URL: https://github.com/apache/nifi/pull/6192#issuecomment-1183739895
Hello,
I really like this new feature and i think it will be very useful!
Is it possible to add support for FetchSmbFile functionality also?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] tpalfy commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
tpalfy commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r926576803
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
Review Comment:
```suggestion
+ "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ "smb://HOSTNAME:PORT/SHARE:\\DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
Review Comment:
```suggestion
"The timestamp of when the file's content ?DID WHAT? in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
Review Comment:
```suggestion
"The minimum age that a file must be in order to be listed; any file younger than this amount of time will be ignored.")
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,370 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"samba, smb, cifs, files", "list"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Lists concrete files shared via SMB protocol. " +
+ "Each listed file may result in one flowfile, the metadata being written as flowfile attributes. " +
+ "Or - in case the 'Record Writer' property is set - the entire result is written as records to a single flowfile. "
+ +
+ "This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the "
+ +
+ "previous node left off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + "to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use Windows's "
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger than the given value will be omitted. Ideally this value should be greater then "
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Age")
+ .name("max-file-age")
+ .description("Any file older than the given value will be omitted. ")
+ .required(false)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
Review Comment:
```suggestion
.description("Any file smaller than the given value will be omitted.")
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
Review Comment:
```suggestion
.description("The network folder from which files should be listed. This is the remaining relative path " +
"after the share: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ " to add subdirectories. The given path on the remote file share must exist. "
+ "This can be checked using verification. You may mix Windows and Linux-style "
+ "directory separators.")
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,370 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"samba, smb, cifs, files", "list"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Lists concrete files shared via SMB protocol. " +
+ "Each listed file may result in one flowfile, the metadata being written as flowfile attributes. " +
+ "Or - in case the 'Record Writer' property is set - the entire result is written as records to a single flowfile. "
+ +
+ "This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the "
+ +
+ "previous node left off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + "to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use Windows's "
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger than the given value will be omitted. Ideally this value should be greater then "
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Age")
+ .name("max-file-age")
+ .description("Any file older than the given value will be omitted. ")
+ .required(false)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
Review Comment:
```suggestion
.description("Any file bigger than the given value will be omitted.")
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,370 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"samba, smb, cifs, files", "list"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Lists concrete files shared via SMB protocol. " +
+ "Each listed file may result in one flowfile, the metadata being written as flowfile attributes. " +
+ "Or - in case the 'Record Writer' property is set - the entire result is written as records to a single flowfile. "
+ +
+ "This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the "
+ +
+ "previous node left off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + "to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use Windows's "
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger than the given value will be omitted. Ideally this value should be greater then "
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Age")
+ .name("max-file-age")
+ .description("Any file older than the given value will be omitted. ")
+ .required(false)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CLIENT_PROVIDER_SERVICE = new Builder()
+ .name("smb-client-provider-service")
+ .displayName("SMB Client Provider Service")
+ .description("Specifies the SMB client provider to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbClientProviderService.class)
+ .build();
+
+ public static final PropertyDescriptor FILE_NAME_SUFFIX_FILTER = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File Name Suffix Filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
Review Comment:
```suggestion
.description("Files ending with the given suffix will be omitted. Can be used to make sure that files "
+ "that are still uploading are not listed multiple times, by having those files have a suffix "
+ "and remove the suffix once the upload finishes. This is highly recommended when using "
+ "'Tracking Entities' or 'Tracking Timestamps' listing strategies.")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] kulikg commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
kulikg commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r921842384
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/NiFiSmbClient.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.stream.StreamSupport.stream;
+
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msfscc.fileinformation.FileIdBothDirectoryInformation;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMBApiException;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.Directory;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.stream.Stream;
+
+public class NiFiSmbClient {
+
+ private static final List<String> SPECIAL_DIRECTORIES = asList(".", "..");
Review Comment:
. and .. are special inode mappings on linux/mac. This is why I gave this name to this variable. Any folder could be hidden. So I'm not sure if HIDDEN_DIRECTORIES is a better name. Using that name we would read in the code:
filter(this::hiddenDirectories) which would make me think we filter every hidden directory which is not the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] samuelwilliams commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
samuelwilliams commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r927129225
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,370 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"samba, smb, cifs, files", "list"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Lists concrete files shared via SMB protocol. " +
+ "Each listed file may result in one flowfile, the metadata being written as flowfile attributes. " +
+ "Or - in case the 'Record Writer' property is set - the entire result is written as records to a single flowfile. "
+ +
+ "This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the "
+ +
+ "previous node left off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + "to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use Windows's "
Review Comment:
Should read: "If so, NiFi will unify all of them and will use the Windows directory separator: '\\'"
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,370 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"samba, smb, cifs, files", "list"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Lists concrete files shared via SMB protocol. " +
+ "Each listed file may result in one flowfile, the metadata being written as flowfile attributes. " +
+ "Or - in case the 'Record Writer' property is set - the entire result is written as records to a single flowfile. "
+ +
+ "This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the "
+ +
+ "previous node left off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + "to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use Windows's "
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger than the given value will be omitted. Ideally this value should be greater then "
Review Comment:
"greater than"
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,370 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"samba, smb, cifs, files", "list"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Lists concrete files shared via SMB protocol. " +
+ "Each listed file may result in one flowfile, the metadata being written as flowfile attributes. " +
+ "Or - in case the 'Record Writer' property is set - the entire result is written as records to a single flowfile. "
+ +
+ "This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the "
+ +
+ "previous node left off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
Review Comment:
Should read: "The network folder from which to list files."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r921491676
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
Review Comment:
This expression language support is not correct, as implemented, expression language is not evaluated, so this line should be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r921492624
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
Review Comment:
Is there a reason to have `Guest` as the default value as opposed to leaving this blank as an optional property?
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(false)
Review Comment:
It looks like this property should be required, correct?
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5")
+ .addValidator(INTEGER_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ PORT,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private Integer port;
Review Comment:
Recommend changing `port` to a primitive `int` since it is a required property.
```suggestion
private int port;
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5")
+ .addValidator(INTEGER_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ PORT,
Review Comment:
Recommend moving `PORT` right after `HOSTNAME`:
```suggestion
HOSTNAME,
PORT,
DOMAIN,
USERNAME,
PASSWORD,
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5")
+ .addValidator(INTEGER_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ PORT,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private Integer port;
+
+ @Override
+ public Session getSession() {
+ try {
+ return smbClient.connect(hostname, port).authenticate(authenticationContext);
Review Comment:
Although chaining these methods is concise, it would be helpful to declare the `Connection` variable and then call `authenticate()` to make it easier to distinguish between connection and authentication failures.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5")
+ .addValidator(INTEGER_VALIDATOR)
Review Comment:
Timeout settings should use the `TIME_PERIOD_VALIDATOR`, which allows various expressions for milliseconds, seconds, or minutes.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
Review Comment:
With renaming the interface, recommend renaming this class to `StandardSmbSessionProviderService` since the interface itself ties the implementation to `SMBJ`.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5")
+ .addValidator(INTEGER_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ PORT,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private Integer port;
+
+ @Override
+ public Session getSession() {
+ try {
+ return smbClient.connect(hostname, port).authenticate(authenticationContext);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
Review Comment:
Recommend including a message with this exception:
```suggestion
throw new UncheckedIOException("SMB connect failed", e);
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5")
+ .addValidator(INTEGER_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ PORT,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private Integer port;
+
+ @Override
+ public Session getSession() {
+ try {
+ return smbClient.connect(hostname, port).authenticate(authenticationContext);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+
+ @Override
+ public URI getServiceLocation() {
+ return URI.create(String.format("smb://%s:%d", hostname, port));
+ }
+
+ @OnEnabled
+ public void onEnabled(ConfigurationContext context) throws IOException {
+ this.context = context;
+ this.hostname = context.getProperty(HOSTNAME).getValue();
+ this.port = context.getProperty(PORT).asInteger();
+ this.smbClient = new SMBClient(SmbConfig.builder()
+ .withTimeout(context.getProperty(TIMEOUT).asLong(), SECONDS)
+ .withTransportLayerFactory(new AsyncDirectTcpTransportFactory<>())
+ .build());
+ createAuthenticationContext();
+ }
+
+ @OnDisabled
+ public void onDisabled() {
+ smbClient.close();
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Collection<ValidationResult> customValidate(ValidationContext validationContext) {
+ final String hostname = validationContext.getProperty(HOSTNAME).getValue();
+ final String port = validationContext.getProperty(PORT).getValue();
+ final String timeout = validationContext.getProperty(TIMEOUT).getValue();
+
+ return asList(HOSTNAME.validate(hostname, validationContext),
+ PORT.validate(port, validationContext),
+ TIMEOUT.validate(timeout, validationContext)
+ );
+ }
Review Comment:
This method can be removed since the standard validators work for each property.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/pom.xml:
##########
@@ -0,0 +1,72 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-smb-connection-pool</artifactId>
Review Comment:
Recommend renaming this module to `nifi-smb-session-service`
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
Review Comment:
```suggestion
@CapabilityDescription("Provides access to SMB Sessions with shared authentication credentials.")
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/pom.xml:
##########
@@ -0,0 +1,72 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-smb-connection-pool</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-connection-pool-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-distributed-cache-client-service-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record-serialization-service-api</artifactId>
+ </dependency>
Review Comment:
These dependencies do not appear to be used.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbSessionProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age in milliseconds")
Review Comment:
The time period can be specified using milliseconds, seconds, minutes, or hours, so this should be changed
```suggestion
.displayName("Minimum File Age")
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbSessionProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age in milliseconds")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5ms")
Review Comment:
```suggestion
.defaultValue("5 ms")
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool-api/pom.xml:
##########
@@ -0,0 +1,39 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>nifi-smb-connection-pool-api</artifactId>
+ <packaging>jar</packaging>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>com.hierynomus</groupId>
+ <artifactId>smbj</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>net.engio</groupId>
+ <artifactId>mbassador</artifactId>
+ </dependency>
Review Comment:
This appears to be a transitive dependency of `smbj` and does not need to be declared directly.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbSessionProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age in milliseconds")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5ms")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size in bytes")
Review Comment:
Corresponding to change the size validation, this can be changed:
```suggestion
.displayName("Minimum File Size")
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbSessionProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age in milliseconds")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5ms")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size in bytes")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size in bytes")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
+ .required(false)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-connection-pool-service")
+ .displayName("SMB Connection Pool Service")
+ .description("Specifies the SMB Connection Pool to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbSessionProviderService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File Name Suffix Filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
+ .required(false)
Review Comment:
It seems like `Share` should be a required property. Is it possible to connect without a Share?
```suggestion
.required(true)
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbSessionProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age in milliseconds")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5ms")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size in bytes")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size in bytes")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
+ .required(false)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
Review Comment:
```suggestion
.addValidator(DATA_SIZE_VALIDATOR)
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/NiFiSmbClientFactory.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+
+public class NiFiSmbClientFactory {
+
+ NiFiSmbClient create(Session session, String shareName) {
+ final Share share = session.connectShare(shareName);
+ if (share instanceof DiskShare) {
+ return new NiFiSmbClient(session, (DiskShare) share);
+ } else {
+ throw new IllegalArgumentException("NiFi supports only disk shares but " +
Review Comment:
Recommend adjusting the wording:
```suggestion
throw new IllegalArgumentException("SMB DiskShare not found. Share " +
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/NiFiSmbClientFactory.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+
+public class NiFiSmbClientFactory {
Review Comment:
Is there a reason for a separate class as opposed to just placing the method in `ListSmb` processor?
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbSessionProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age in milliseconds")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5ms")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size in bytes")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size in bytes")
Review Comment:
```suggestion
.displayName("Maximum File Size")
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5")
Review Comment:
Changing to use the Time Period Validator will allow changing this to `5 secs`.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool-api/pom.xml:
##########
@@ -0,0 +1,39 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>nifi-smb-connection-pool-api</artifactId>
Review Comment:
With the interface change to `SmbSessionProviderService`, recommend renaming the module to `nifi-smb-session-api`, and renaming the NAR to `nifi-smb-session-api-nar`.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/pom.xml:
##########
@@ -0,0 +1,72 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-smb-connection-pool</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-connection-pool-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-distributed-cache-client-service-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record-serialization-service-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.hierynomus</groupId>
+ <artifactId>smbj</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>net.engio</groupId>
+ <artifactId>mbassador</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-lang3</artifactId>
+ </dependency>
Review Comment:
These dependencies do not appear to be used.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbSessionProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age in milliseconds")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5ms")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size in bytes")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size in bytes")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
+ .required(false)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-connection-pool-service")
+ .displayName("SMB Connection Pool Service")
+ .description("Specifies the SMB Connection Pool to use for creating SMB connections.")
Review Comment:
This should be adjusted based on the interface renaming:
```suggestion
.name("smb-session-provider-service")
.displayName("SMB Session Provider Service")
.description("Specifies the SMB Session Provider to use for obtaining SMB connections.")
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/NiFiSmbClient.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.stream.StreamSupport.stream;
+
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msfscc.fileinformation.FileIdBothDirectoryInformation;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMBApiException;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.Directory;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.stream.Stream;
+
+public class NiFiSmbClient {
Review Comment:
Renaming this class would help communicate the purpose. Perhaps `SmbSessionClient` or `SmbShareClient`?
Also seeing that this class is being mocked for testing, recommend defining it as an interface and then creating a standard implementation.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbSessionProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age in milliseconds")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5ms")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size in bytes")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
Review Comment:
This should be changed to use the `DATA_SIZE_VALIDATOR`, which supports various expressions.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/NiFiSmbClientFactory.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+
+public class NiFiSmbClientFactory {
+
+ NiFiSmbClient create(Session session, String shareName) {
+ final Share share = session.connectShare(shareName);
+ if (share instanceof DiskShare) {
+ return new NiFiSmbClient(session, (DiskShare) share);
+ } else {
+ throw new IllegalArgumentException("NiFi supports only disk shares but " +
+ share.getClass().getSimpleName() + " found on host " + session.getConnection().getRemoteHostname()
+ + "!");
Review Comment:
```suggestion
share.getClass().getSimpleName() + " found on host " + session.getConnection().getRemoteHostname());
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/NiFiSmbClient.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.stream.StreamSupport.stream;
+
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msfscc.fileinformation.FileIdBothDirectoryInformation;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMBApiException;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.Directory;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.stream.Stream;
+
+public class NiFiSmbClient {
+
+ private static final List<String> SPECIAL_DIRECTORIES = asList(".", "..");
Review Comment:
Perhaps renaming this `HIDDEN_DIRECTORIES` would be better?
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/NiFiSmbClient.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.stream.StreamSupport.stream;
+
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msfscc.fileinformation.FileIdBothDirectoryInformation;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMBApiException;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.Directory;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.stream.Stream;
+
+public class NiFiSmbClient {
+
+ private static final List<String> SPECIAL_DIRECTORIES = asList(".", "..");
+
+ private final Session session;
+ private final DiskShare share;
+
+ NiFiSmbClient(Session session, DiskShare share) {
+ this.session = session;
+ this.share = share;
+ }
+
+ static String unifyDirectorySeparators(String path) {
+ return path.replace('/', '\\');
+ }
+
+ public void close() {
+ try {
+ session.close();
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
Review Comment:
An exception message should be added:
```suggestion
throw new UncheckedIOException("SMB Session close failed", e);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] kulikg commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
kulikg commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r923879170
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client-provider/src/main/java/org/apache/nifi/services/smb/NiFiSmbjClient.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.stream.StreamSupport.stream;
+
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msfscc.fileinformation.FileIdBothDirectoryInformation;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMBApiException;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.Directory;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.nifi.services.smb.NiFiSmbClient;
+import org.apache.nifi.services.smb.SmbListableEntity;
+
+public class NiFiSmbjClient implements NiFiSmbClient {
Review Comment:
Thank you for your help again! Isn't calling it simply SmbjClientService name good enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r923884830
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client-provider/src/main/java/org/apache/nifi/services/smb/NiFiSmbjClient.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.stream.StreamSupport.stream;
+
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msfscc.fileinformation.FileIdBothDirectoryInformation;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMBApiException;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.Directory;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.nifi.services.smb.NiFiSmbClient;
+import org.apache.nifi.services.smb.SmbListableEntity;
+
+public class NiFiSmbjClient implements NiFiSmbClient {
Review Comment:
Thanks, in this case, `SmbjClientService` sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] kulikg commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
kulikg commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r920794378
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool-api/src/main/java/org/apache/nifi/services/smb/SmbConnectionPoolService.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import org.apache.nifi.controller.ControllerService;
+
+public interface SmbConnectionPoolService extends ControllerService {
+
+ /**
+ * Returns the name of the share to connect.
+ *
+ * @return the share
+ */
+ String getShareName();
+
+ /**
+ * Returns the hostname to connect to.
+ *
+ * @return the hostname
+ */
+ String getHostname();
+
+ /**
+ * Returns the port using to connect.
+ *
+ * @return the port.
+ */
+ Integer getPort();
+
+ /**
+ * Returns the SmbClient to use
+ *
+ * @return the smbClient
+ */
+ SMBClient getSmbClient();
+
+ /**
+ * Returns the authentication context.
+ *
+ * @return the authentication context.
+ */
+ AuthenticationContext getAuthenticationContext();
Review Comment:
Thank you for your help making this processor better! We've discussed your suggestions and decided to share the session from the interface. I really like the idea to have a generic file listing interface. But it would turn ListSmb processor into a generic listing processor and would be a bit confusing next to the other listing processors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r921137902
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbConnectionPoolService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file in the filesystem was modified as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
Review Comment:
I would consider to add Maximum File Age and Minimum/Maximum File Size proprties too (as in case of some other List processors).
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbConnectionPoolService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file in the filesystem was modified as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum file age in milliseconds")
Review Comment:
Please use `StandardValidators.TIME_PERIOD_VALIDATOR` for time periods with arbitrary time units instead of milliseconds only.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbConnectionPoolService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file in the filesystem was modified as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum file age in milliseconds")
+ .name("min-age")
+ .description(
+ "Any file younger the the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(INTEGER_VALIDATOR)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5000")
+ .build();
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-connection-pool-service")
+ .displayName("SMB Connection Pool Service")
+ .description("Specifies the SMB Connection Pool to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbConnectionPoolService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File name suffix filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ AbstractListProcessor.RECORD_WRITER,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET,
+ SMB_LISTING_STRATEGY,
+ SMB_CONNECTION_POOL_SERVICE,
+ DIRECTORY,
+ MINIMUM_AGE,
+ SKIP_FILES_WITH_SUFFIX,
+ SHARE
Review Comment:
Please order the properties from most important to least important.
Suggested order:
- SMB_LISTING_STRATEGY
- SMB_CONNECTION_POOL_SERVICE
- SHARE
- DIRECTORY
- AbstractListProcessor.RECORD_WRITER
- SKIP_FILES_WITH_SUFFIX
- MINIMUM_AGE
- AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION
- ListedEntityTracker.TRACKING_STATE_CACHE
- ListedEntityTracker.TRACKING_TIME_WINDOW
- ListedEntityTracker.INITIAL_LISTING_TARGET
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool-api/src/main/java/org/apache/nifi/services/smb/SmbConnectionPoolService.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import com.hierynomus.smbj.session.Session;
+import java.net.URI;
+import org.apache.nifi.controller.ControllerService;
+
+public interface SmbConnectionPoolService extends ControllerService {
+
+ /**
+ * Returns the identifier of the service location.
+ *
+ * @return the remote location
+ */
+ URI getServiceLocation();
+
+ /**
+ * Returns the active session to use.
+ *
+ * @return the session.
+ */
+ Session getSession();
Review Comment:
Returning `Session` object from the controller service seems to me fine. However, in that case it should be renamed to `SmbSessionProvider` or something like that.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbConnectionPoolService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file in the filesystem was modified as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum file age in milliseconds")
+ .name("min-age")
+ .description(
+ "Any file younger the the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(INTEGER_VALIDATOR)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5000")
+ .build();
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-connection-pool-service")
+ .displayName("SMB Connection Pool Service")
+ .description("Specifies the SMB Connection Pool to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbConnectionPoolService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File name suffix filter")
Review Comment:
Please use title case for property labels: "File Name Suffix Filter"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r922296919
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5")
+ .addValidator(INTEGER_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ PORT,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private Integer port;
+
+ @Override
+ public Session getSession() {
+ try {
+ return smbClient.connect(hostname, port).authenticate(authenticationContext);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+
+ @Override
+ public URI getServiceLocation() {
+ return URI.create(String.format("smb://%s:%d", hostname, port));
+ }
+
+ @OnEnabled
+ public void onEnabled(ConfigurationContext context) throws IOException {
+ this.context = context;
+ this.hostname = context.getProperty(HOSTNAME).getValue();
+ this.port = context.getProperty(PORT).asInteger();
+ this.smbClient = new SMBClient(SmbConfig.builder()
+ .withTimeout(context.getProperty(TIMEOUT).asLong(), SECONDS)
+ .withTransportLayerFactory(new AsyncDirectTcpTransportFactory<>())
+ .build());
+ createAuthenticationContext();
+ }
+
+ @OnDisabled
+ public void onDisabled() {
+ smbClient.close();
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Collection<ValidationResult> customValidate(ValidationContext validationContext) {
+ final String hostname = validationContext.getProperty(HOSTNAME).getValue();
+ final String port = validationContext.getProperty(PORT).getValue();
+ final String timeout = validationContext.getProperty(TIMEOUT).getValue();
+
+ return asList(HOSTNAME.validate(hostname, validationContext),
+ PORT.validate(port, validationContext),
+ TIMEOUT.validate(timeout, validationContext)
+ );
+ }
Review Comment:
Yes, the `customValidate()` method works in addition to the validators defined on each PropertyDescriptor, so it is not necessary to implement `customValidate()` when standard validators are defined. A custom validation implementation can be used to evaluate dependencies between properties, or other logic, but the standard validators work when a property can be evaluate on its own.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] kulikg commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
kulikg commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r926544342
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-client-provider-service")
+ .displayName("SMB Client Provider Service")
+ .description("Specifies the SMB client provider to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbClientProviderService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File Name Suffix Filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ SMB_LISTING_STRATEGY,
+ SMB_CONNECTION_POOL_SERVICE,
+ DIRECTORY,
+ AbstractListProcessor.RECORD_WRITER,
+ SKIP_FILES_WITH_SUFFIX,
+ MINIMUM_AGE,
+ MINIMUM_SIZE,
+ MAXIMUM_SIZE,
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET
+ ));
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Map<String, String> createAttributes(SmbListableEntity entity, ProcessContext context) {
+ final Map<String, String> attributes = new TreeMap<>();
+ attributes.put("filename", entity.getName());
+ attributes.put("shortname", entity.getShortName());
+ attributes.put("path", entity.getPath());
+ attributes.put("absolute.path", getPath(context) + entity.getPathWithName());
+ attributes.put("identifier", entity.getIdentifier());
+ attributes.put("timestamp",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getTimestamp(), 0, ZoneOffset.UTC)));
+ attributes.put("creationTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getCreationTime(), 0, ZoneOffset.UTC)));
+ attributes.put("lastAccessedTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getLastAccessTime(), 0, ZoneOffset.UTC)));
+ attributes.put("changeTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getChangeTime(), 0, ZoneOffset.UTC)));
+ attributes.put("size", String.valueOf(entity.getSize()));
+ attributes.put("allocationSize", String.valueOf(entity.getAllocationSize()));
+ return unmodifiableMap(attributes);
+ }
+
+ @Override
+ protected String getPath(ProcessContext context) {
+ final SmbClientProviderService connectionPoolService =
+ context.getProperty(SMB_CONNECTION_POOL_SERVICE).asControllerService(SmbClientProviderService.class);
+ final URI serviceLocation = connectionPoolService.getServiceLocation();
+ final String directory = getDirectory(context);
+ return String.format("%s:\\%s", serviceLocation.toString(), directory.isEmpty() ? "" : directory + "\\");
+ }
+
+ @Override
+ protected List<SmbListableEntity> performListing(ProcessContext context, Long minimumTimestampOrNull,
+ ListingMode listingMode) throws IOException {
+
+ final Predicate<SmbListableEntity> fileFilter =
+ createFileFilter(context, minimumTimestampOrNull);
+
+ try (Stream<SmbListableEntity> listing = performListing(context)) {
+ final Iterator<SmbListableEntity> iterator = listing.iterator();
+ final List<SmbListableEntity> result = new LinkedList<>();
+ while (iterator.hasNext()) {
+ if (!isExecutionScheduled(listingMode)) {
+ return emptyList();
+ }
+ final SmbListableEntity entity = iterator.next();
+ if (fileFilter.test(entity)) {
+ result.add(entity);
+ }
+ }
+ return result;
Review Comment:
Please note that a processor can be stopped during listing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] samuelwilliams commented on pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
samuelwilliams commented on PR #6192:
URL: https://github.com/apache/nifi/pull/6192#issuecomment-1184407412
It might be worthwhile placing additional attributes on the flowfiles. You can get the following attributes from the `FileIdBothDirectoryInformation` class:
* `creationTime`
* `lastAccessTime`
* `lastWriteTime`
* `changeTime`
* `endOfFile`
* `allocationSize`
* `shortName`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r918189322
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjConnectionPoolService.java:
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.util.StringUtils;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjConnectionPoolService extends AbstractControllerService implements SmbConnectionPoolService {
Review Comment:
It would be helpful to provide a property for configuring the Timeout on SMBClient connections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] kulikg commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
kulikg commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r921856706
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5")
+ .addValidator(INTEGER_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ PORT,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private Integer port;
+
+ @Override
+ public Session getSession() {
+ try {
+ return smbClient.connect(hostname, port).authenticate(authenticationContext);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+
+ @Override
+ public URI getServiceLocation() {
+ return URI.create(String.format("smb://%s:%d", hostname, port));
+ }
+
+ @OnEnabled
+ public void onEnabled(ConfigurationContext context) throws IOException {
+ this.context = context;
+ this.hostname = context.getProperty(HOSTNAME).getValue();
+ this.port = context.getProperty(PORT).asInteger();
+ this.smbClient = new SMBClient(SmbConfig.builder()
+ .withTimeout(context.getProperty(TIMEOUT).asLong(), SECONDS)
+ .withTransportLayerFactory(new AsyncDirectTcpTransportFactory<>())
+ .build());
+ createAuthenticationContext();
+ }
+
+ @OnDisabled
+ public void onDisabled() {
+ smbClient.close();
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Collection<ValidationResult> customValidate(ValidationContext validationContext) {
+ final String hostname = validationContext.getProperty(HOSTNAME).getValue();
+ final String port = validationContext.getProperty(PORT).getValue();
+ final String timeout = validationContext.getProperty(TIMEOUT).getValue();
+
+ return asList(HOSTNAME.validate(hostname, validationContext),
+ PORT.validate(port, validationContext),
+ TIMEOUT.validate(timeout, validationContext)
+ );
+ }
Review Comment:
By adding this I override this function from AbstractConfigurableComponent:
protected Collection<ValidationResult> customValidate(final ValidationContext validationContext) {
return Collections.emptySet();
}
Are you sure it will properly validate during enabling without this function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] samuelwilliams commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
samuelwilliams commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r921146614
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/NiFiSmbClient.java:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.stream.StreamSupport.stream;
+
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msfscc.fileinformation.FileIdBothDirectoryInformation;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMBApiException;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.Directory;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.stream.Stream;
+
+public class NiFiSmbClient {
+
+ private static final List<String> SPECIAL_DIRECTORIES = asList(".", "..");
+
+ private final Session session;
+ private final DiskShare share;
+
+ NiFiSmbClient(Session session, DiskShare share) {
+ this.session = session;
+ this.share = share;
+ }
+
+ static String unifyDirectorySeparators(String path) {
+ return path.replace('/', '\\');
+ }
+
+ public void close() {
+ try {
+ session.close();
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+
+ Stream<SmbListableEntity> listRemoteFiles(SmbListableEntity listable) {
+ return listRemoteFiles(listable.getPathWithName());
+ }
+
+ Stream<SmbListableEntity> listRemoteFiles(String path) {
+ final Directory directory = openDirectory(path);
+ return stream(directory::spliterator, 0, false)
+ .map(entity -> buildSmbListableEntity(entity, path))
+ .filter(this::specialDirectory)
+ .flatMap(listable -> listable.isDirectory() ? listRemoteFiles(listable) : Stream.of(listable))
+ .onClose(directory::close);
+ }
+
+ private SmbListableEntity buildSmbListableEntity(FileIdBothDirectoryInformation info, String path) {
Review Comment:
It would be useful to also grab the `creationTime`, `lastAccessTime`, and `shortName` as attributes on the flowfile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] kulikg commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
kulikg commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r926525477
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientProviderService.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides access to SMB Sessions with shared authentication credentials.")
+public class SmbjClientProviderService extends AbstractControllerService implements SmbClientProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5 secs")
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ PORT,
+ SHARE,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private String share;
+ private int port;
+
+ @Override
+ public SmbClientService getClient() {
+ try {
+ final Connection connection = smbClient.connect(hostname, port);
+ final Session session = connection.authenticate(authenticationContext);
+ final String shareName = context.getProperty(SHARE).getValue();
+ final Share share = session.connectShare(shareName);
+ if (share instanceof DiskShare) {
+ return new SmbjClientService(session, (DiskShare) share);
+ } else {
+ throw new IllegalArgumentException("DiskShare not found. Share " +
+ share.getClass().getSimpleName() + " found on host " + session.getConnection()
+ .getRemoteHostname());
Review Comment:
It would look something like: "DiskShare not found. PrinterShare found on host ..."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r928315803
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientProviderService.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"samba, smb, cifs, files"})
+@CapabilityDescription("Provides access to SMB Sessions with shared authentication credentials.")
+public class SmbjClientProviderService extends AbstractControllerService implements SmbClientProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout for read and write operations.")
+ .required(true)
+ .defaultValue("5 secs")
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ PORT,
+ SHARE,
+ USERNAME,
+ PASSWORD,
+ DOMAIN,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private String hostname;
+ private int port;
+ private String shareName;
+
+ @Override
+ public SmbClientService getClient() {
+ try {
+ final Connection connection = smbClient.connect(hostname, port);
+ final Session session = connection.authenticate(authenticationContext);
Review Comment:
@kulikg It seems the connection can become stale and unusable under some circumstances. In these cases, `connection.isConnected()` still returns `true` (that's why `SMBClient` does not create a new one) but `connection.authenticate(...)` fails.
I ran into the following cases:
1. Using SMB server running in a Docker container locally. After 60 seconds (even when runnig in every 2 seconds, so no inactivity):
```
com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
at com.hierynomus.smbj.connection.SMBSessionBuilder.establish(SMBSessionBuilder.java:114)
at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:202)
at org.apache.nifi.services.smb.SmbjClientProviderService.getClient(SmbjClientProviderService.java:126)
```
2. Using a Windows share. After some idle / inactive time:
```
com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: java.net.SocketException: Broken pipe (Write failed)
at com.hierynomus.smbj.connection.SMBSessionBuilder.establish(SMBSessionBuilder.java:114)
at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:202)
at org.apache.nifi.services.smb.SmbjClientProviderService.getClient(SmbjClientProviderService.java:126)
```
It seems to be a bug in the smbj library or some extra config may help.
As a workaround, the exceptions above could be caught and the connection could be closed forcefully and then `SMBClient.connect()` would create a new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r926313390
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-client-api/src/main/java/org/apache/nifi/services/smb/SmbListableEntity.java:
##########
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import org.apache.nifi.processor.util.list.ListableEntity;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+
+public class SmbListableEntity implements ListableEntity {
+
+ private final String name;
+ private final String shortName;
+ private final String path;
+ private final long timestamp;
+ private final long creationTime;
+ private final long lastAccessTime;
+ private final long changeTime;
+ private final boolean directory;
+ private final long size;
+ private final long allocationSize;
+
+ private SmbListableEntity(String name, String shortName, String path, long timestamp, long creationTime,
+ long lastAccessTime, long changeTime, boolean directory,
+ long size, long allocationSize) {
+ this.name = name;
+ this.shortName = shortName;
+ this.path = path;
+ this.timestamp = timestamp;
+ this.creationTime = creationTime;
+ this.lastAccessTime = lastAccessTime;
+ this.changeTime = changeTime;
+ this.directory = directory;
+ this.size = size;
+ this.allocationSize = allocationSize;
+ }
+
+ public static SimpleRecordSchema getRecordSchema() {
+ List<RecordField> fields = Arrays.asList(
+ new RecordField("name", RecordFieldType.STRING.getDataType(), false),
+ new RecordField("shortName", RecordFieldType.STRING.getDataType(), false),
+ new RecordField("path", RecordFieldType.STRING.getDataType(), false),
+ new RecordField("identifier", RecordFieldType.STRING.getDataType(), false),
+ new RecordField("timeStamp", RecordFieldType.LONG.getDataType(), false),
+ new RecordField("createTime", RecordFieldType.LONG.getDataType(), false),
+ new RecordField("lastAccessTime", RecordFieldType.LONG.getDataType(), false),
+ new RecordField("changeTime", RecordFieldType.LONG.getDataType(), false),
+ new RecordField("size", RecordFieldType.LONG.getDataType(), false),
+ new RecordField("allocationSize", RecordFieldType.LONG.getDataType(), false)
+ );
+ return new SimpleRecordSchema(fields);
+ }
+
+ public static SmbListableEntityBuilder builder() {
+ return new SmbListableEntityBuilder();
+ }
+
+ public String getShortName() {
+ return shortName;
+ }
+
+ public long getCreationTime() {
+ return creationTime;
+ }
+
+ public long getLastAccessTime() {
+ return lastAccessTime;
+ }
+
+ public long getChangeTime() {
+ return changeTime;
+ }
+
+ public long getAllocationSize() {
+ return allocationSize;
+ }
+
+ @Override
+ public String getName() {
+ return name;
+ }
+
+ public String getPath() {
+ return path;
+ }
+
+ public String getPathWithName() {
+ return path.isEmpty() ? name : path + "\\" + name;
+ }
+
+ @Override
+ public String getIdentifier() {
+ return getPathWithName();
+ }
+
+ @Override
+ public long getTimestamp() {
+ return timestamp;
+ }
+
+ @Override
+ public long getSize() {
+ return size;
+ }
+
+ public boolean isDirectory() {
+ return directory;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ SmbListableEntity that = (SmbListableEntity) o;
+ return getPathWithName().equals(that.getPathWithName());
+ }
+
+ @Override
+ public int hashCode() {
+ return getPathWithName().hashCode();
+ }
+
+ @Override
+ public String toString() {
+ return getPathWithName() + " (last write: " + timestamp + " size: " + size + ")";
+ }
+
+ @Override
+ public Record toRecord() {
+ final Map<String, Object> record = new TreeMap<>();
+ record.put("name", name);
+ record.put("path", path);
+ record.put("identifier", getPathWithName());
+ record.put("timestamp", getTimestamp());
+ record.put("size", size);
+ return new MapRecord(getRecordSchema(), record);
Review Comment:
The record schema declares more fields. Should not all of them be initialized here?
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-client-api/src/main/java/org/apache/nifi/services/smb/SmbClientService.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import java.io.OutputStream;
+import java.util.stream.Stream;
+
+/**
+ * Service abstraction for Server Message Block protocol operations.
+ */
+public interface SmbClientService {
+
+ Stream<SmbListableEntity> listRemoteFiles(String path);
+
+ void createDirectory(String path);
+
+ OutputStream getOutputStreamForFile(String path);
+
+ void close();
Review Comment:
The interface should be defined as `AutoClosable` for using it in try-with-resources (if some client code would prefer that).
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-client-api/src/main/java/org/apache/nifi/services/smb/SmbListableEntity.java:
##########
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import org.apache.nifi.processor.util.list.ListableEntity;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+
+public class SmbListableEntity implements ListableEntity {
+
+ private final String name;
+ private final String shortName;
+ private final String path;
+ private final long timestamp;
+ private final long creationTime;
+ private final long lastAccessTime;
+ private final long changeTime;
+ private final boolean directory;
+ private final long size;
+ private final long allocationSize;
+
+ private SmbListableEntity(String name, String shortName, String path, long timestamp, long creationTime,
+ long lastAccessTime, long changeTime, boolean directory,
+ long size, long allocationSize) {
+ this.name = name;
+ this.shortName = shortName;
+ this.path = path;
+ this.timestamp = timestamp;
+ this.creationTime = creationTime;
+ this.lastAccessTime = lastAccessTime;
+ this.changeTime = changeTime;
+ this.directory = directory;
+ this.size = size;
+ this.allocationSize = allocationSize;
+ }
+
+ public static SimpleRecordSchema getRecordSchema() {
+ List<RecordField> fields = Arrays.asList(
+ new RecordField("name", RecordFieldType.STRING.getDataType(), false),
+ new RecordField("shortName", RecordFieldType.STRING.getDataType(), false),
+ new RecordField("path", RecordFieldType.STRING.getDataType(), false),
+ new RecordField("identifier", RecordFieldType.STRING.getDataType(), false),
+ new RecordField("timeStamp", RecordFieldType.LONG.getDataType(), false),
+ new RecordField("createTime", RecordFieldType.LONG.getDataType(), false),
+ new RecordField("lastAccessTime", RecordFieldType.LONG.getDataType(), false),
+ new RecordField("changeTime", RecordFieldType.LONG.getDataType(), false),
+ new RecordField("size", RecordFieldType.LONG.getDataType(), false),
+ new RecordField("allocationSize", RecordFieldType.LONG.getDataType(), false)
Review Comment:
I suggest using the same names for the record fields as for the FlowFile attributes:
- name => filename
- timeStamp => timestamp
- createTime => creationTime
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/pom.xml:
##########
@@ -0,0 +1,65 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-smbj-client</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-client-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-distributed-cache-client-service-api</artifactId>
+ </dependency>
Review Comment:
Not used dependency.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientProviderService.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
Review Comment:
I suggest using similar list of tags as in case of the existing `PutSmb` processor:
`{"samba, smb, cifs, files"}`
Not sure if "microsoft" is needed.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/pom.xml:
##########
@@ -0,0 +1,65 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-smbj-client</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-client-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-distributed-cache-client-service-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.hierynomus</groupId>
+ <artifactId>smbj</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
Review Comment:
Not used dependency.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientProviderService.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides access to SMB Sessions with shared authentication credentials.")
+public class SmbjClientProviderService extends AbstractControllerService implements SmbClientProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
Review Comment:
Typo: no space before "after".
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientProviderService.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides access to SMB Sessions with shared authentication credentials.")
+public class SmbjClientProviderService extends AbstractControllerService implements SmbClientProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5 secs")
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ PORT,
+ SHARE,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private String share;
+ private int port;
+
+ @Override
+ public SmbClientService getClient() {
+ try {
+ final Connection connection = smbClient.connect(hostname, port);
+ final Session session = connection.authenticate(authenticationContext);
+ final String shareName = context.getProperty(SHARE).getValue();
Review Comment:
The share name is available as a field (line 120). Not needed to retrieve it from the context.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientProviderService.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides access to SMB Sessions with shared authentication credentials.")
+public class SmbjClientProviderService extends AbstractControllerService implements SmbClientProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5 secs")
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ PORT,
+ SHARE,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private String share;
+ private int port;
+
+ @Override
+ public SmbClientService getClient() {
+ try {
+ final Connection connection = smbClient.connect(hostname, port);
+ final Session session = connection.authenticate(authenticationContext);
+ final String shareName = context.getProperty(SHARE).getValue();
+ final Share share = session.connectShare(shareName);
+ if (share instanceof DiskShare) {
+ return new SmbjClientService(session, (DiskShare) share);
+ } else {
+ throw new IllegalArgumentException("DiskShare not found. Share " +
+ share.getClass().getSimpleName() + " found on host " + session.getConnection()
+ .getRemoteHostname());
+ }
+ } catch (IOException e) {
+ throw new UncheckedIOException("Could not connect to " + hostname, e);
+ }
+ }
+
+ @Override
+ public URI getServiceLocation() {
+ return URI.create(String.format("smb://%s:%d/%s", hostname, port, share));
+ }
+
+ @OnEnabled
+ public void onEnabled(ConfigurationContext context) throws IOException {
+ this.context = context;
Review Comment:
It is not needed to store the `context`. It can simply be passed in `createAuthenticationContext()`.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientProviderService.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides access to SMB Sessions with shared authentication credentials.")
+public class SmbjClientProviderService extends AbstractControllerService implements SmbClientProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5 secs")
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ PORT,
+ SHARE,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private String share;
+ private int port;
Review Comment:
Could you please move `port` up in order to follow the logical order: `host, port, share`?
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientProviderService.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides access to SMB Sessions with shared authentication credentials.")
+public class SmbjClientProviderService extends AbstractControllerService implements SmbClientProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5 secs")
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ PORT,
+ SHARE,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private String share;
+ private int port;
+
+ @Override
+ public SmbClientService getClient() {
+ try {
+ final Connection connection = smbClient.connect(hostname, port);
+ final Session session = connection.authenticate(authenticationContext);
+ final String shareName = context.getProperty(SHARE).getValue();
+ final Share share = session.connectShare(shareName);
+ if (share instanceof DiskShare) {
+ return new SmbjClientService(session, (DiskShare) share);
+ } else {
+ throw new IllegalArgumentException("DiskShare not found. Share " +
+ share.getClass().getSimpleName() + " found on host " + session.getConnection()
+ .getRemoteHostname());
Review Comment:
The second sentence says "Share xxx **found** on host zzz". It should be **"not found"**, should not it?
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientProviderService.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides access to SMB Sessions with shared authentication credentials.")
+public class SmbjClientProviderService extends AbstractControllerService implements SmbClientProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5 secs")
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ PORT,
+ SHARE,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
Review Comment:
The `DOMAIN` property's description says: "Optional, in most cases username and password is sufficient."
For this reason, I would consider to move it after `USERNAME/PASSWORD`.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientProviderService.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides access to SMB Sessions with shared authentication credentials.")
+public class SmbjClientProviderService extends AbstractControllerService implements SmbClientProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5 secs")
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ PORT,
+ SHARE,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private String share;
+ private int port;
+
+ @Override
+ public SmbClientService getClient() {
+ try {
+ final Connection connection = smbClient.connect(hostname, port);
+ final Session session = connection.authenticate(authenticationContext);
+ final String shareName = context.getProperty(SHARE).getValue();
+ final Share share = session.connectShare(shareName);
+ if (share instanceof DiskShare) {
+ return new SmbjClientService(session, (DiskShare) share);
+ } else {
+ throw new IllegalArgumentException("DiskShare not found. Share " +
+ share.getClass().getSimpleName() + " found on host " + session.getConnection()
+ .getRemoteHostname());
+ }
+ } catch (IOException e) {
+ throw new UncheckedIOException("Could not connect to " + hostname, e);
+ }
+ }
+
+ @Override
+ public URI getServiceLocation() {
+ return URI.create(String.format("smb://%s:%d/%s", hostname, port, share));
+ }
+
+ @OnEnabled
+ public void onEnabled(ConfigurationContext context) throws IOException {
+ this.context = context;
+ this.hostname = context.getProperty(HOSTNAME).getValue();
+ this.share = context.getProperty(SHARE).getValue();
+ this.port = context.getProperty(PORT).asInteger();
+ this.smbClient = new SMBClient(SmbConfig.builder()
+ .withTimeout(context.getProperty(TIMEOUT).asTimePeriod(SECONDS), SECONDS)
+ .build());
+ createAuthenticationContext();
+ }
+
+ @OnDisabled
+ public void onDisabled() {
+ smbClient.close();
Review Comment:
The fields should be set to `null` too.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientProviderService.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides access to SMB Sessions with shared authentication credentials.")
+public class SmbjClientProviderService extends AbstractControllerService implements SmbClientProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
+ .required(true)
+ .defaultValue("5 secs")
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ PORT,
+ SHARE,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private String share;
+ private int port;
+
+ @Override
+ public SmbClientService getClient() {
+ try {
+ final Connection connection = smbClient.connect(hostname, port);
+ final Session session = connection.authenticate(authenticationContext);
+ final String shareName = context.getProperty(SHARE).getValue();
+ final Share share = session.connectShare(shareName);
+ if (share instanceof DiskShare) {
+ return new SmbjClientService(session, (DiskShare) share);
+ } else {
+ throw new IllegalArgumentException("DiskShare not found. Share " +
+ share.getClass().getSimpleName() + " found on host " + session.getConnection()
+ .getRemoteHostname());
+ }
+ } catch (IOException e) {
+ throw new UncheckedIOException("Could not connect to " + hostname, e);
+ }
+ }
+
+ @Override
+ public URI getServiceLocation() {
+ return URI.create(String.format("smb://%s:%d/%s", hostname, port, share));
+ }
+
+ @OnEnabled
+ public void onEnabled(ConfigurationContext context) throws IOException {
+ this.context = context;
+ this.hostname = context.getProperty(HOSTNAME).getValue();
+ this.share = context.getProperty(SHARE).getValue();
+ this.port = context.getProperty(PORT).asInteger();
Review Comment:
Could you please move `port` up in order to follow the logical order: `host, port, share`?
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
Review Comment:
Typos:
- extra space before "to add" in line 117
- no space before "directory" in line 120
- not sure but `Windows' ...` would be correct
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-client-provider-service")
+ .displayName("SMB Client Provider Service")
+ .description("Specifies the SMB client provider to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbClientProviderService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File Name Suffix Filter")
Review Comment:
Please rename the constant too.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
Review Comment:
I suggest using similar list of tags as in case of the existing PutSmb processor:
{"samba, smb, cifs, files", "list"}
Not sure if "microsoft" is needed.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-client-provider-service")
+ .displayName("SMB Client Provider Service")
+ .description("Specifies the SMB client provider to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbClientProviderService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File Name Suffix Filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ SMB_LISTING_STRATEGY,
+ SMB_CONNECTION_POOL_SERVICE,
+ DIRECTORY,
+ AbstractListProcessor.RECORD_WRITER,
+ SKIP_FILES_WITH_SUFFIX,
+ MINIMUM_AGE,
+ MINIMUM_SIZE,
+ MAXIMUM_SIZE,
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET
+ ));
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Map<String, String> createAttributes(SmbListableEntity entity, ProcessContext context) {
+ final Map<String, String> attributes = new TreeMap<>();
+ attributes.put("filename", entity.getName());
+ attributes.put("shortname", entity.getShortName());
+ attributes.put("path", entity.getPath());
+ attributes.put("absolute.path", getPath(context) + entity.getPathWithName());
+ attributes.put("identifier", entity.getIdentifier());
+ attributes.put("timestamp",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getTimestamp(), 0, ZoneOffset.UTC)));
+ attributes.put("creationTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getCreationTime(), 0, ZoneOffset.UTC)));
+ attributes.put("lastAccessedTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getLastAccessTime(), 0, ZoneOffset.UTC)));
+ attributes.put("changeTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getChangeTime(), 0, ZoneOffset.UTC)));
+ attributes.put("size", String.valueOf(entity.getSize()));
+ attributes.put("allocationSize", String.valueOf(entity.getAllocationSize()));
+ return unmodifiableMap(attributes);
+ }
+
+ @Override
+ protected String getPath(ProcessContext context) {
+ final SmbClientProviderService connectionPoolService =
Review Comment:
Please rename the variable too.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientProviderService.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides access to SMB Sessions with shared authentication credentials.")
+public class SmbjClientProviderService extends AbstractControllerService implements SmbClientProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout in seconds for read and write operations.")
Review Comment:
"in seconds" should be deleted because any time unit can be used.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-client-provider-service")
+ .displayName("SMB Client Provider Service")
+ .description("Specifies the SMB client provider to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbClientProviderService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File Name Suffix Filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ SMB_LISTING_STRATEGY,
+ SMB_CONNECTION_POOL_SERVICE,
+ DIRECTORY,
+ AbstractListProcessor.RECORD_WRITER,
+ SKIP_FILES_WITH_SUFFIX,
+ MINIMUM_AGE,
+ MINIMUM_SIZE,
+ MAXIMUM_SIZE,
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET
+ ));
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Map<String, String> createAttributes(SmbListableEntity entity, ProcessContext context) {
+ final Map<String, String> attributes = new TreeMap<>();
+ attributes.put("filename", entity.getName());
+ attributes.put("shortname", entity.getShortName());
+ attributes.put("path", entity.getPath());
+ attributes.put("absolute.path", getPath(context) + entity.getPathWithName());
+ attributes.put("identifier", entity.getIdentifier());
+ attributes.put("timestamp",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getTimestamp(), 0, ZoneOffset.UTC)));
+ attributes.put("creationTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getCreationTime(), 0, ZoneOffset.UTC)));
+ attributes.put("lastAccessedTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getLastAccessTime(), 0, ZoneOffset.UTC)));
+ attributes.put("changeTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getChangeTime(), 0, ZoneOffset.UTC)));
Review Comment:
I can see weird values in the timestamps for the year: `+54496-06-21T20:14:18`
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-client-provider-service")
+ .displayName("SMB Client Provider Service")
+ .description("Specifies the SMB client provider to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbClientProviderService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File Name Suffix Filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ SMB_LISTING_STRATEGY,
+ SMB_CONNECTION_POOL_SERVICE,
+ DIRECTORY,
+ AbstractListProcessor.RECORD_WRITER,
+ SKIP_FILES_WITH_SUFFIX,
+ MINIMUM_AGE,
+ MINIMUM_SIZE,
+ MAXIMUM_SIZE,
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET
+ ));
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Map<String, String> createAttributes(SmbListableEntity entity, ProcessContext context) {
+ final Map<String, String> attributes = new TreeMap<>();
+ attributes.put("filename", entity.getName());
+ attributes.put("shortname", entity.getShortName());
+ attributes.put("path", entity.getPath());
+ attributes.put("absolute.path", getPath(context) + entity.getPathWithName());
+ attributes.put("identifier", entity.getIdentifier());
+ attributes.put("timestamp",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getTimestamp(), 0, ZoneOffset.UTC)));
+ attributes.put("creationTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getCreationTime(), 0, ZoneOffset.UTC)));
+ attributes.put("lastAccessedTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getLastAccessTime(), 0, ZoneOffset.UTC)));
+ attributes.put("changeTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getChangeTime(), 0, ZoneOffset.UTC)));
+ attributes.put("size", String.valueOf(entity.getSize()));
+ attributes.put("allocationSize", String.valueOf(entity.getAllocationSize()));
+ return unmodifiableMap(attributes);
+ }
+
+ @Override
+ protected String getPath(ProcessContext context) {
+ final SmbClientProviderService connectionPoolService =
+ context.getProperty(SMB_CONNECTION_POOL_SERVICE).asControllerService(SmbClientProviderService.class);
+ final URI serviceLocation = connectionPoolService.getServiceLocation();
+ final String directory = getDirectory(context);
+ return String.format("%s:\\%s", serviceLocation.toString(), directory.isEmpty() ? "" : directory + "\\");
+ }
+
+ @Override
+ protected List<SmbListableEntity> performListing(ProcessContext context, Long minimumTimestampOrNull,
+ ListingMode listingMode) throws IOException {
+
+ final Predicate<SmbListableEntity> fileFilter =
+ createFileFilter(context, minimumTimestampOrNull);
+
+ try (Stream<SmbListableEntity> listing = performListing(context)) {
+ final Iterator<SmbListableEntity> iterator = listing.iterator();
+ final List<SmbListableEntity> result = new LinkedList<>();
+ while (iterator.hasNext()) {
+ if (!isExecutionScheduled(listingMode)) {
+ return emptyList();
+ }
+ final SmbListableEntity entity = iterator.next();
+ if (fileFilter.test(entity)) {
+ result.add(entity);
+ }
+ }
+ return result;
Review Comment:
Having a `Stream` at hand, could not foreach + collect be used?
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-client-provider-service")
+ .displayName("SMB Client Provider Service")
+ .description("Specifies the SMB client provider to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbClientProviderService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File Name Suffix Filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ SMB_LISTING_STRATEGY,
+ SMB_CONNECTION_POOL_SERVICE,
+ DIRECTORY,
+ AbstractListProcessor.RECORD_WRITER,
+ SKIP_FILES_WITH_SUFFIX,
+ MINIMUM_AGE,
+ MINIMUM_SIZE,
+ MAXIMUM_SIZE,
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET
+ ));
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Map<String, String> createAttributes(SmbListableEntity entity, ProcessContext context) {
+ final Map<String, String> attributes = new TreeMap<>();
+ attributes.put("filename", entity.getName());
+ attributes.put("shortname", entity.getShortName());
+ attributes.put("path", entity.getPath());
+ attributes.put("absolute.path", getPath(context) + entity.getPathWithName());
+ attributes.put("identifier", entity.getIdentifier());
+ attributes.put("timestamp",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getTimestamp(), 0, ZoneOffset.UTC)));
+ attributes.put("creationTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getCreationTime(), 0, ZoneOffset.UTC)));
+ attributes.put("lastAccessedTime",
Review Comment:
Suggestion: `lastAccessedTime` => `lastAccessTime` (as in case of the record writer approach)
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-client-provider-service")
+ .displayName("SMB Client Provider Service")
+ .description("Specifies the SMB client provider to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbClientProviderService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File Name Suffix Filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ SMB_LISTING_STRATEGY,
+ SMB_CONNECTION_POOL_SERVICE,
+ DIRECTORY,
+ AbstractListProcessor.RECORD_WRITER,
+ SKIP_FILES_WITH_SUFFIX,
+ MINIMUM_AGE,
+ MINIMUM_SIZE,
+ MAXIMUM_SIZE,
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET
+ ));
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Map<String, String> createAttributes(SmbListableEntity entity, ProcessContext context) {
+ final Map<String, String> attributes = new TreeMap<>();
+ attributes.put("filename", entity.getName());
+ attributes.put("shortname", entity.getShortName());
+ attributes.put("path", entity.getPath());
+ attributes.put("absolute.path", getPath(context) + entity.getPathWithName());
+ attributes.put("identifier", entity.getIdentifier());
+ attributes.put("timestamp",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getTimestamp(), 0, ZoneOffset.UTC)));
+ attributes.put("creationTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getCreationTime(), 0, ZoneOffset.UTC)));
+ attributes.put("lastAccessedTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getLastAccessTime(), 0, ZoneOffset.UTC)));
+ attributes.put("changeTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getChangeTime(), 0, ZoneOffset.UTC)));
+ attributes.put("size", String.valueOf(entity.getSize()));
+ attributes.put("allocationSize", String.valueOf(entity.getAllocationSize()));
+ return unmodifiableMap(attributes);
+ }
+
+ @Override
+ protected String getPath(ProcessContext context) {
+ final SmbClientProviderService connectionPoolService =
+ context.getProperty(SMB_CONNECTION_POOL_SERVICE).asControllerService(SmbClientProviderService.class);
+ final URI serviceLocation = connectionPoolService.getServiceLocation();
+ final String directory = getDirectory(context);
+ return String.format("%s:\\%s", serviceLocation.toString(), directory.isEmpty() ? "" : directory + "\\");
+ }
+
+ @Override
+ protected List<SmbListableEntity> performListing(ProcessContext context, Long minimumTimestampOrNull,
+ ListingMode listingMode) throws IOException {
+
+ final Predicate<SmbListableEntity> fileFilter =
+ createFileFilter(context, minimumTimestampOrNull);
+
+ try (Stream<SmbListableEntity> listing = performListing(context)) {
+ final Iterator<SmbListableEntity> iterator = listing.iterator();
+ final List<SmbListableEntity> result = new LinkedList<>();
+ while (iterator.hasNext()) {
+ if (!isExecutionScheduled(listingMode)) {
+ return emptyList();
+ }
+ final SmbListableEntity entity = iterator.next();
+ if (fileFilter.test(entity)) {
+ result.add(entity);
+ }
+ }
+ return result;
+ } catch (Exception e) {
+ throw new IOException("Could not perform listing", e);
+ }
+ }
+
+ @Override
+ protected boolean isListingResetNecessary(PropertyDescriptor property) {
+ return asList(SMB_CONNECTION_POOL_SERVICE, DIRECTORY, SKIP_FILES_WITH_SUFFIX).contains(property);
+ }
+
+ @Override
+ protected Scope getStateScope(PropertyContext context) {
+ return CLUSTER;
+ }
+
+ @Override
+ protected RecordSchema getRecordSchema() {
+ return SmbListableEntity.getRecordSchema();
+ }
+
+ @Override
+ protected Integer countUnfilteredListing(ProcessContext context) throws IOException {
+ try (Stream<SmbListableEntity> listing = performListing(context)) {
+ return Long.valueOf(listing.count()).intValue();
+ } catch (Exception e) {
+ throw new IOException("Could not count files", e);
+ }
+ }
+
+ @Override
+ protected String getListingContainerName(ProcessContext context) {
+ return String.format("Remote Directory [%s]", getPath(context));
+ }
+
+ private boolean isExecutionScheduled(ListingMode listingMode) {
+ return ListingMode.CONFIGURATION_VERIFICATION.equals(listingMode) || isScheduled();
+ }
+
+ private Predicate<SmbListableEntity> createFileFilter(ProcessContext context, Long minTimestampOrNull) {
+
+ final Long minimumAge = context.getProperty(MINIMUM_AGE).asTimePeriod(TimeUnit.MILLISECONDS);
+ final Double minimumSizeOrNull =
+ context.getProperty(MINIMUM_SIZE).isSet() ? context.getProperty(MINIMUM_SIZE).asDataSize(DataUnit.B) : null;
+ final Double maximumSizeOrNull =
+ context.getProperty(MAXIMUM_SIZE).isSet() ? context.getProperty(MAXIMUM_SIZE).asDataSize(DataUnit.B) : null;
+ final String suffixOrNull = context.getProperty(SKIP_FILES_WITH_SUFFIX).getValue();
+
+ final long now = getCurrentTime();
+ Predicate<SmbListableEntity> filter = entity -> now - entity.getTimestamp() >= minimumAge;
+
+ if (minTimestampOrNull != null) {
+ filter = filter.and(entity -> entity.getTimestamp() >= minTimestampOrNull);
+ }
+
+ if (minimumSizeOrNull != null) {
+ filter = filter.and(entity -> entity.getSize() >= minimumSizeOrNull);
+ }
+
+ if (maximumSizeOrNull != null) {
+ filter = filter.and(entity -> entity.getSize() <= maximumSizeOrNull);
+ }
+
+ if (suffixOrNull != null) {
+ filter = filter.and(entity -> !entity.getName().endsWith(suffixOrNull));
+ }
+
+ return filter;
+ }
+
+ private Stream<SmbListableEntity> performListing(ProcessContext context) throws IOException {
+ final SmbClientProviderService connectionPoolService =
Review Comment:
Please rename the variable too.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
Review Comment:
There is also another type of usage: use the Record Writer property and add the listing as a json in the FlowFile content.
Could please add the same (or similar) description as in case of the recently added ListGoogleDrive?
https://github.com/apache/nifi/blob/ad781170e31682ddb9be1c3dc906e758c5909ed4/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/drive/ListGoogleDrive.java#L72-L76
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientService.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.stream.StreamSupport.stream;
+
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msfscc.fileinformation.FileIdBothDirectoryInformation;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMBApiException;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.Directory;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.stream.Stream;
+
+public class SmbjClientService implements SmbClientService {
+
+ private static final List<String> SPECIAL_DIRECTORIES = asList(".", "..");
+
+ private final Session session;
+ private final DiskShare share;
+
+ public SmbjClientService(Session session, DiskShare share) {
+ this.session = session;
+ this.share = share;
+ }
+
+ @Override
+ public void close() {
+ try {
+ session.close();
+ } catch (IOException e) {
+ throw new UncheckedIOException("SMB Session close failed", e);
+ }
+ }
+
+ @Override
+ public Stream<SmbListableEntity> listRemoteFiles(String path) {
+ final Directory directory = openDirectory(path);
+ return stream(directory::spliterator, 0, false)
+ .map(entity -> buildSmbListableEntity(entity, path))
+ .filter(this::specialDirectory)
+ .flatMap(listable -> listable.isDirectory() ? listRemoteFiles(listable) : Stream.of(listable))
+ .onClose(directory::close);
+ }
+
+ @Override
+ public void createDirectory(String path) {
+ final int lastDirectorySeparatorPosition = path.lastIndexOf("\\");
+ if (lastDirectorySeparatorPosition > 0) {
+ createDirectory(path.substring(0, lastDirectorySeparatorPosition));
+ }
+ if (!share.folderExists(path)) {
+ share.mkdir(path);
+ }
+ }
+
+ @Override
+ public OutputStream getOutputStreamForFile(String path) {
+ try {
+ final File file = share.openFile(
+ path,
+ EnumSet.of(AccessMask.GENERIC_WRITE),
+ EnumSet.of(FileAttributes.FILE_ATTRIBUTE_NORMAL),
+ EnumSet.of(SMB2ShareAccess.FILE_SHARE_READ, SMB2ShareAccess.FILE_SHARE_WRITE,
+ SMB2ShareAccess.FILE_SHARE_DELETE),
+ SMB2CreateDisposition.FILE_OVERWRITE_IF,
+ EnumSet.of(SMB2CreateOptions.FILE_WRITE_THROUGH));
+ final OutputStream smbOutputStream = file.getOutputStream();
+ return new OutputStream() {
+ @Override
+ public void write(int b) throws IOException {
+ smbOutputStream.write(b);
+ }
+
+ @Override
+ public void flush() throws IOException {
+ smbOutputStream.flush();
+ }
+
+ @Override
+ public void close() throws IOException {
+ smbOutputStream.close();
+ file.close();
+ }
+ };
+ } catch (SMBApiException s) {
+ throw new RuntimeException("Could not open " + path + " for writing", s);
+ }
+ }
+
+ private Stream<SmbListableEntity> listRemoteFiles(SmbListableEntity listable) {
+ return listRemoteFiles(listable.getPathWithName());
+ }
+
+ private SmbListableEntity buildSmbListableEntity(FileIdBothDirectoryInformation info, String path) {
+ return SmbListableEntity.builder()
+ .setName(info.getFileName())
+ .setShortName(info.getShortName())
+ .setPath(path)
+ .setTimestamp(info.getLastWriteTime().toEpochMillis())
+ .setCreationTime(info.getCreationTime().toEpochMillis())
+ .setChangeTime(info.getChangeTime().toEpochMillis())
+ .setLastAccessTime(info.getLastAccessTime().toEpochMillis())
+ .setDirectory((info.getFileAttributes() & FileAttributes.FILE_ATTRIBUTE_DIRECTORY.getValue()) != 0)
+ .setSize(info.getEndOfFile())
+ .setAllocationSize(info.getAllocationSize())
+ .build();
+ }
+
+ private Directory openDirectory(String path) {
+ try {
+ return share.openDirectory(
+ path,
+ EnumSet.of(AccessMask.GENERIC_READ),
+ EnumSet.of(FileAttributes.FILE_ATTRIBUTE_DIRECTORY),
+ EnumSet.of(SMB2ShareAccess.FILE_SHARE_READ),
+ SMB2CreateDisposition.FILE_OPEN,
+ EnumSet.of(SMB2CreateOptions.FILE_DIRECTORY_FILE)
+ );
+ } catch (SMBApiException s) {
+ throw new RuntimeException("Could not open directory " + path + " due to " + s.getMessage(), s);
+ }
+ }
+
+ private boolean specialDirectory(SmbListableEntity entity) {
+ return !SPECIAL_DIRECTORIES.contains(entity.getName());
+ }
Review Comment:
It should be called `nonSpecialDirectory` (or return the non-negated value.).
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientService.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.stream.StreamSupport.stream;
+
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msfscc.fileinformation.FileIdBothDirectoryInformation;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMBApiException;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.Directory;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.stream.Stream;
+
+public class SmbjClientService implements SmbClientService {
+
+ private static final List<String> SPECIAL_DIRECTORIES = asList(".", "..");
+
+ private final Session session;
+ private final DiskShare share;
+
+ public SmbjClientService(Session session, DiskShare share) {
+ this.session = session;
+ this.share = share;
+ }
+
+ @Override
+ public void close() {
+ try {
+ session.close();
+ } catch (IOException e) {
+ throw new UncheckedIOException("SMB Session close failed", e);
+ }
+ }
+
+ @Override
+ public Stream<SmbListableEntity> listRemoteFiles(String path) {
+ final Directory directory = openDirectory(path);
+ return stream(directory::spliterator, 0, false)
+ .map(entity -> buildSmbListableEntity(entity, path))
+ .filter(this::specialDirectory)
+ .flatMap(listable -> listable.isDirectory() ? listRemoteFiles(listable) : Stream.of(listable))
+ .onClose(directory::close);
+ }
+
+ @Override
+ public void createDirectory(String path) {
+ final int lastDirectorySeparatorPosition = path.lastIndexOf("\\");
+ if (lastDirectorySeparatorPosition > 0) {
+ createDirectory(path.substring(0, lastDirectorySeparatorPosition));
+ }
+ if (!share.folderExists(path)) {
+ share.mkdir(path);
+ }
+ }
+
+ @Override
+ public OutputStream getOutputStreamForFile(String path) {
+ try {
+ final File file = share.openFile(
+ path,
+ EnumSet.of(AccessMask.GENERIC_WRITE),
+ EnumSet.of(FileAttributes.FILE_ATTRIBUTE_NORMAL),
+ EnumSet.of(SMB2ShareAccess.FILE_SHARE_READ, SMB2ShareAccess.FILE_SHARE_WRITE,
+ SMB2ShareAccess.FILE_SHARE_DELETE),
+ SMB2CreateDisposition.FILE_OVERWRITE_IF,
+ EnumSet.of(SMB2CreateOptions.FILE_WRITE_THROUGH));
+ final OutputStream smbOutputStream = file.getOutputStream();
+ return new OutputStream() {
+ @Override
+ public void write(int b) throws IOException {
+ smbOutputStream.write(b);
+ }
+
+ @Override
+ public void flush() throws IOException {
+ smbOutputStream.flush();
+ }
+
+ @Override
+ public void close() throws IOException {
+ smbOutputStream.close();
+ file.close();
+ }
+ };
+ } catch (SMBApiException s) {
+ throw new RuntimeException("Could not open " + path + " for writing", s);
+ }
+ }
+
+ private Stream<SmbListableEntity> listRemoteFiles(SmbListableEntity listable) {
+ return listRemoteFiles(listable.getPathWithName());
+ }
Review Comment:
Why is it a separate method?
If `listRemoteFiles(String)` was called directly, the recursion would be more straightforward and the code more readable.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-client-provider-service")
+ .displayName("SMB Client Provider Service")
Review Comment:
Please rename the constant too.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbConnectionPoolService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file in the filesystem was modified as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
Review Comment:
Is the a specific reason why `Maximum File Age` was not added?
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
Review Comment:
Typos:
- then => than
- no space before "the amount"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r923430649
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-client-provider-api/pom.xml:
##########
@@ -0,0 +1,48 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>nifi-smb-client-provider-api</artifactId>
+ <packaging>jar</packaging>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-listed-entity</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.hierynomus</groupId>
+ <artifactId>smbj</artifactId>
+ </dependency>
Review Comment:
This dependency should be removed from this `api` module.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-client-provider-api/src/main/java/org/apache/nifi/services/smb/NiFiSmbClient.java:
##########
@@ -0,0 +1,15 @@
+package org.apache.nifi.services.smb;
+
+import java.io.OutputStream;
+import java.util.stream.Stream;
+
+public interface NiFiSmbClient {
+
+ Stream<SmbListableEntity> listRemoteFiles(String path);
+
+ void createDirectory(String path);
+
+ OutputStream getOutputStreamForFile(String pathAndFileName);
Review Comment:
Recommend adjusting this method name and parameter:
```suggestion
OutputStream getFileOutputStream(String path);
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client-provider/pom.xml:
##########
@@ -0,0 +1,77 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-smbj-client-provider</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-client-provider-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-distributed-cache-client-service-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record</artifactId>
+ </dependency>
+ <!--dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record-serialization-service-api</artifactId>
+ </dependency-->
Review Comment:
This commented dependency should be removed.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client-provider/pom.xml:
##########
@@ -0,0 +1,77 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-smbj-client-provider</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-client-provider-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-distributed-cache-client-service-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record</artifactId>
+ </dependency>
+ <!--dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record-serialization-service-api</artifactId>
+ </dependency-->
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.hierynomus</groupId>
+ <artifactId>smbj</artifactId>
+ </dependency>
+ <!--dependency>
+ <groupId>net.engio</groupId>
+ <artifactId>mbassador</artifactId>
+ </dependency-->
Review Comment:
This commented dependency should be removed.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-client-provider-api/src/main/java/org/apache/nifi/services/smb/NiFiSmbClient.java:
##########
@@ -0,0 +1,15 @@
+package org.apache.nifi.services.smb;
+
+import java.io.OutputStream;
+import java.util.stream.Stream;
+
+public interface NiFiSmbClient {
Review Comment:
Although the `NiFi` prefix helps to distinguish this interface from the SMBJ `SmbClient`, prefixing components with `NiFi` is not a great pattern because the package name already indicates NiFi association. What about naming this `SmbClientService`?
```suggestion
public interface SmbClientService {
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-client-provider-api/src/main/java/org/apache/nifi/services/smb/NiFiSmbClient.java:
##########
@@ -0,0 +1,15 @@
+package org.apache.nifi.services.smb;
+
+import java.io.OutputStream;
+import java.util.stream.Stream;
+
Review Comment:
It would be helpful to produce a basic description of this interface, just a sentence or two should be sufficient.
```suggestion
/**
* Service abstraction for Server Message Block protocol operations
*/
```
##########
nifi-assembly/pom.xml:
##########
@@ -664,6 +664,18 @@ language governing permissions and limitations under the License. -->
<version>1.17.0-SNAPSHOT</version>
<type>nar</type>
</dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-connection-pool-api-nar</artifactId>
Review Comment:
This should be changed to `nifi-smb-client-provider-api-nar`
##########
nifi-assembly/pom.xml:
##########
@@ -664,6 +664,18 @@ language governing permissions and limitations under the License. -->
<version>1.17.0-SNAPSHOT</version>
<type>nar</type>
</dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-connection-pool-api-nar</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ <type>nar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-connection-pool-nar</artifactId>
Review Comment:
This should be changed to `nifi-smb-client-provider-nar`
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/pom.xml:
##########
@@ -26,23 +26,62 @@
<packaging>jar</packaging>
<dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-client-provider-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ </dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-distributed-cache-client-service-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record-serialization-service-api</artifactId>
+ </dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-utils</artifactId>
- <version>1.17.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.hierynomus</groupId>
<artifactId>smbj</artifactId>
- <version>0.10.0</version>
+ </dependency>
+ <dependency>
+ <groupId>net.engio</groupId>
+ <artifactId>mbassador</artifactId>
+ </dependency>
Review Comment:
This dependency appears unnecessary.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-client-provider-api/pom.xml:
##########
@@ -0,0 +1,48 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>nifi-smb-client-provider-api</artifactId>
+ <packaging>jar</packaging>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-listed-entity</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.hierynomus</groupId>
+ <artifactId>smbj</artifactId>
+ </dependency>
+ <!--dependency>
+ <groupId>net.engio</groupId>
+ <artifactId>mbassador</artifactId>
+ </dependency-->
Review Comment:
Commented dependency should be removed
```suggestion
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client-provider/pom.xml:
##########
@@ -0,0 +1,77 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-smbj-client-provider</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-client-provider-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-distributed-cache-client-service-api</artifactId>
+ </dependency>
Review Comment:
This dependency does not appear to be used.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client-provider/src/main/java/org/apache/nifi/services/smb/NiFiSmbjClient.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.stream.StreamSupport.stream;
+
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msfscc.fileinformation.FileIdBothDirectoryInformation;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMBApiException;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.Directory;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.nifi.services.smb.NiFiSmbClient;
+import org.apache.nifi.services.smb.SmbListableEntity;
+
+public class NiFiSmbjClient implements NiFiSmbClient {
Review Comment:
Following the naming recommend to avoid prefixing with `NiFi`, recommend naming this class `SmbjSmbClientService` or `StandardSmbClientService`.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/pom.xml:
##########
@@ -26,23 +26,62 @@
<packaging>jar</packaging>
<dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-client-provider-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ </dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-distributed-cache-client-service-api</artifactId>
+ </dependency>
Review Comment:
This dependency appears to be unnecessary.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smbj-client-provider/pom.xml:
##########
@@ -0,0 +1,77 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-smbj-client-provider</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-client-provider-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-distributed-cache-client-service-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record</artifactId>
+ </dependency>
+ <!--dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record-serialization-service-api</artifactId>
+ </dependency-->
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.hierynomus</groupId>
+ <artifactId>smbj</artifactId>
+ </dependency>
+ <!--dependency>
+ <groupId>net.engio</groupId>
+ <artifactId>mbassador</artifactId>
+ </dependency-->
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <!--dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-lang3</artifactId>
+ </dependency-->
Review Comment:
This commented dependency should be removed.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/test/java/org/apache/nifi/processors/smb/ListSmbTest.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.util.Arrays.stream;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static org.apache.nifi.processor.util.list.AbstractListProcessor.LISTING_STRATEGY;
+import static org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+import static org.apache.nifi.processor.util.list.AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION;
+import static org.apache.nifi.processor.util.list.ListedEntityTracker.TRACKING_STATE_CACHE;
+import static org.apache.nifi.processors.smb.ListSmb.DIRECTORY;
+import static org.apache.nifi.processors.smb.ListSmb.MINIMUM_AGE;
+import static org.apache.nifi.processors.smb.ListSmb.SKIP_FILES_WITH_SUFFIX;
+import static org.apache.nifi.processors.smb.ListSmb.SMB_CONNECTION_POOL_SERVICE;
+import static org.apache.nifi.util.TestRunners.newTestRunner;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.processor.util.list.ListedEntity;
+import org.apache.nifi.services.smb.NiFiSmbClient;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+import org.apache.nifi.util.TestRunner;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class ListSmbTest {
+
+ private final static AtomicLong currentMillis = new AtomicLong();
+ private final static AtomicLong currentNanos = new AtomicLong();
+
+ private static long currentMillis() {
+ return currentMillis.get();
+ }
+
+ private static long currentNanos() {
+ return currentNanos.get();
+ }
+
+ private static void setTime(Long timeInMillis) {
+ currentMillis.set(timeInMillis);
+ currentNanos.set(NANOSECONDS.convert(timeInMillis, MILLISECONDS));
+ }
+
+ private static void timePassed(Long timeInMillis) {
+ currentMillis.addAndGet(timeInMillis);
+ currentNanos.addAndGet(NANOSECONDS.convert(timeInMillis, MILLISECONDS));
+ }
+
+ @Test
+ public void shouldResetStateWhenPropertiesChanged() throws Exception {
+ final TestRunner testRunner = newTestRunner(ListSmb.class);
+ testRunner.setProperty(LISTING_STRATEGY, "timestamps");
+ testRunner.setProperty(TARGET_SYSTEM_TIMESTAMP_PRECISION, "millis");
+ testRunner.setProperty(MINIMUM_AGE, "0 ms");
+ final NiFiSmbClient mockNifiSmbClient = configureTestRunnerWithMockedSambaClient(testRunner);
+ long now = System.currentTimeMillis();
+ mockSmbFolders(mockNifiSmbClient, listableEntity("should_list_it_after_each_reset", now - 100));
+ testRunner.run();
+ testRunner.assertTransferCount(REL_SUCCESS, 1);
+ testRunner.setProperty(DIRECTORY, "testDirectoryChanged");
+ testRunner.run();
+ testRunner.assertTransferCount(REL_SUCCESS, 2);
+
+ testRunner.setProperty(SKIP_FILES_WITH_SUFFIX, "suffix_changed");
+ testRunner.run();
+ testRunner.assertTransferCount(REL_SUCCESS, 3);
+
+ final SmbClientProviderService connectionPoolService = mock(SmbClientProviderService.class);
+ when(connectionPoolService.getIdentifier()).thenReturn("connection-pool-2");
+ when(connectionPoolService.getServiceLocation()).thenReturn(URI.create("smb://localhost:445/share"));
+ when(connectionPoolService.getClient()).thenReturn(mockNifiSmbClient);
+// final NiFiSmbClientFactory mockSmbClientFactory = mock(NiFiSmbClientFactory.class);
+// ((ListSmb) testRunner.getProcessor()).smbClientFactory = mockSmbClientFactory;
+// when(mockSmbClientFactory.create(any(), any())).thenReturn(mockNifiSmbClient);
Review Comment:
These commented lines should be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] samuelwilliams commented on pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
samuelwilliams commented on PR #6192:
URL: https://github.com/apache/nifi/pull/6192#issuecomment-1184913854
@kulikg - Did you have the intention of refactoring the existing GetSmb and PutSmb processors to utilise the client? Or do you think this is a body of work for later?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r922291708
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
Review Comment:
It would be great if we could isolate the SMBJ specifics to the Controller Service implementation, and avoid having interface-level dependency on SMBJ. Thanks for working these adjustments!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] kulikg commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
kulikg commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r921839944
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
Review Comment:
in the auth context you have 2 options to authenticate without any credentials:
AuthenticationContext.anonymous();
AuthenticationContext.guest();
guest() will authenticate with the "Guest" user and empty password. This is one way to keep this convention. I can remove it if you'd like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r918153432
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/pom.xml:
##########
@@ -0,0 +1,72 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-bundle</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-smb-connection-pool</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-connection-pool-api</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-distributed-cache-client-service-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record-serialization-service-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.hierynomus</groupId>
+ <artifactId>smbj</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>net.engio</groupId>
+ <artifactId>mbassador</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>commons-lang</groupId>
+ <artifactId>commons-lang</artifactId>
Review Comment:
Apache Commons-Lang3 should be used instead of the older Apache Commons-Lang.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool-api/src/main/java/org/apache/nifi/services/smb/SmbConnectionPoolService.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import org.apache.nifi.controller.ControllerService;
+
+public interface SmbConnectionPoolService extends ControllerService {
+
+ /**
+ * Returns the name of the share to connect.
+ *
+ * @return the share
+ */
+ String getShareName();
+
+ /**
+ * Returns the hostname to connect to.
+ *
+ * @return the hostname
+ */
+ String getHostname();
+
+ /**
+ * Returns the port using to connect.
+ *
+ * @return the port.
+ */
+ Integer getPort();
+
+ /**
+ * Returns the SmbClient to use
+ *
+ * @return the smbClient
+ */
+ SMBClient getSmbClient();
+
+ /**
+ * Returns the authentication context.
+ *
+ * @return the authentication context.
+ */
+ AuthenticationContext getAuthenticationContext();
Review Comment:
Instead of returning these smbj objects, access could be encapsulated in several ways. One option is to encapsulate the authentication process and return a `Session`, which would allow reuse. A better option would be to encapsulate the service operation and provide a method that would enumerate a shared directory along the following lines of:
```
List<ListableEntity> getRemoteFiles(String directory);
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool-api/src/main/java/org/apache/nifi/services/smb/SmbConnectionPoolService.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import org.apache.nifi.controller.ControllerService;
+
+public interface SmbConnectionPoolService extends ControllerService {
+
+ /**
+ * Returns the name of the share to connect.
+ *
+ * @return the share
+ */
+ String getShareName();
+
+ /**
+ * Returns the hostname to connect to.
+ *
+ * @return the hostname
+ */
+ String getHostname();
+
+ /**
+ * Returns the port using to connect.
+ *
+ * @return the port.
+ */
+ Integer getPort();
Review Comment:
Instead of exposing individual property values, the service interface could be simplified to return `java.net.URI` that would encapsulate SMB URI construction.
```suggestion
/**
* Get the service location including the protocol, hostname, and port for the current connection
*
* @return Service Location URI
*/
URI getServiceLocation();
```
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+
+import java.io.IOException;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbConnectionPoolService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file in the filesystem was modified as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum file age in milliseconds")
+ .name("min-age")
+ .description(
+ "Any file younger the the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(INTEGER_VALIDATOR)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5000")
+ .build();
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-connection-pool-service")
+ .displayName("SMB Connection Pool Service")
+ .description("Specifies the SMB Connection Pool to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbConnectionPoolService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("skip-files-with-suffix")
+ .displayName("File name suffix filter")
Review Comment:
As this is a new property, it would be best to align the property name and display name. Should it be `file-name-suffix-filter`? Taking a closer look at the approach, is there a reason for limiting the scope to the filename suffix, as opposed to supporting a more generalized Filename Filter property?
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjConnectionPoolService.java:
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.util.StringUtils;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjConnectionPoolService extends AbstractControllerService implements SmbConnectionPoolService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host to which files should be written.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be written. This is the \"first folder\"" +
+ "after the hostname: \\\\hostname\\[share]\\dir1\\dir2")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication. If no username is set then anonymous authentication is attempted.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication. Required if Username is set.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(INTEGER_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(Arrays.asList(
+ HOSTNAME,
+ SHARE,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ PORT
+ ));
+
+ public SMBClient getSmbClient() {
+ return smbClient;
+ }
+
+ public AuthenticationContext getAuthenticationContext() {
+ return authenticationContext;
+ }
+
+ private final SMBClient smbClient = new SMBClient();
+ private AuthenticationContext authenticationContext;
+ private ConfigurationContext context;
+ private String hostname;
+ private Integer port;
+ private String shareName;
+
+ public String getShareName() {
+ return shareName;
+ }
+
+ public String getHostname() {
+ return hostname;
+ }
+
+ public Integer getPort() {
+ return port;
+ }
+
+ @OnEnabled
+ public void onEnabled(ConfigurationContext context) {
+ this.context = context;
+ this.hostname = context.getProperty(HOSTNAME).getValue();
+ this.port = context.getProperty(PORT).asInteger();
+ this.shareName = context.getProperty(SHARE).getValue();
+ createAuthenticationContext();
+ }
+
+ private void createAuthenticationContext() {
+ final String userName = context.getProperty(USERNAME).getValue();
+ final String password = context.getProperty(PASSWORD).getValue();
+ final String domain = context.getProperty(DOMAIN).getValue();
+ if (userName != null && password != null) {
+ authenticationContext = new AuthenticationContext(userName, password.toCharArray(), domain);
+ } else {
+ authenticationContext = AuthenticationContext.anonymous();
+ }
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Collection<ValidationResult> customValidate(ValidationContext validationContext) {
+ final List<ValidationResult> results = new ArrayList<>();
+
+ final String hostname = validationContext.getProperty(HOSTNAME).getValue();
+ final Integer port = validationContext.getProperty(PORT).asInteger();
+ final String share = validationContext.getProperty(SHARE).getValue();
+
+ if (StringUtils.isBlank(hostname)) {
+ results.add(new ValidationResult.Builder()
+ .subject(this.getClass().getSimpleName())
+ .valid(false)
+ .explanation("hostname is required")
+ .build()
+ );
+ }
+
+ if (StringUtils.isBlank(share)) {
+ results.add(new ValidationResult.Builder()
+ .subject(this.getClass().getSimpleName())
+ .valid(false)
+ .explanation("share is required")
+ .build()
+ );
+ }
+
+ if (port == null || port <= 0) {
+ results.add(new ValidationResult.Builder()
+ .subject(this.getClass().getSimpleName())
+ .valid(false)
+ .explanation("port is invalid")
+ .build()
+ );
+ }
+
+ return results;
+ }
Review Comment:
Is this method necessary? It looks like this validation should be handled using the validators configured for each property descriptor.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+
+import java.io.IOException;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbConnectionPoolService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file in the filesystem was modified as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum file age in milliseconds")
+ .name("min-age")
+ .description(
+ "Any file younger the the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(INTEGER_VALIDATOR)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5000")
+ .build();
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-connection-pool-service")
+ .displayName("SMB Connection Pool Service")
+ .description("Specifies the SMB Connection Pool to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbConnectionPoolService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("skip-files-with-suffix")
+ .displayName("File name suffix filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final String FILE_MODIFY_DATE_ATTR_FORMAT = "yyyy-MM-dd'T'HH:mm:ssZ";
+ private static final DateFormat formatter = new SimpleDateFormat(FILE_MODIFY_DATE_ATTR_FORMAT, Locale.US);
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ AbstractListProcessor.RECORD_WRITER,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET,
+ SMB_LISTING_STRATEGY,
+ SMB_CONNECTION_POOL_SERVICE,
+ DIRECTORY,
+ MINIMUM_AGE,
+ SKIP_FILES_WITH_SUFFIX
+ ));
+
+ NiFiSmbClientFactory smbClientFactory = new NiFiSmbClientFactory();
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Map<String, String> createAttributes(SmbListableEntity entity, ProcessContext context) {
+ final Map<String, String> attributes = new TreeMap<>();
+ attributes.put("filename", entity.getName());
+ attributes.put("path", entity.getPath());
+ attributes.put("absolute.path", getPath(context) + entity.getPathWithName());
+ attributes.put("identifier", entity.getIdentifier());
+ attributes.put("timestamp", formatter.format(new Date(entity.getTimestamp())));
+ attributes.put("size", String.valueOf(entity.getSize()));
+ return unmodifiableMap(attributes);
+ }
+
+ @Override
+ protected String getPath(ProcessContext context) {
+ final SmbConnectionPoolService connectionPoolService =
+ context.getProperty(SMB_CONNECTION_POOL_SERVICE).asControllerService(SmbConnectionPoolService.class);
+ final String hostname = connectionPoolService.getHostname();
+ final String port = String.valueOf(connectionPoolService.getPort());
+ final String share = connectionPoolService.getShareName();
+ final String directory = getDirectory(context);
+ return String.format("smb://%s:%s/%s:\\%s", hostname, port, share, directory.isEmpty() ? "" : directory + "\\");
+ }
+
+ @Override
+ protected List<SmbListableEntity> performListing(ProcessContext context, Long minimumTimestampOrNull,
+ ListingMode listingMode) throws IOException {
+
+ final Integer minimumAge = context.getProperty(MINIMUM_AGE).asInteger();
+ final String suffixOrNull = context.getProperty(SKIP_FILES_WITH_SUFFIX).getValue();
+ final Predicate<SmbListableEntity> fileFilter =
+ createFileFilter(minimumAge, minimumTimestampOrNull, suffixOrNull);
+ try (Stream<SmbListableEntity> listing = performListing(context)) {
+ final Iterator<SmbListableEntity> iterator = listing.iterator();
+ final List<SmbListableEntity> result = new LinkedList<>();
+ while (iterator.hasNext()) {
+ if (!isExecutionScheduled(listingMode)) {
+ return emptyList();
+ }
+ final SmbListableEntity entity = iterator.next();
+ if (fileFilter.test(entity)) {
+ result.add(entity);
+ }
+ }
+ return result;
+ } catch (IOException o) {
+ throw o;
+ } catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+
+ @Override
+ protected boolean isListingResetNecessary(PropertyDescriptor property) {
+ return asList(SMB_CONNECTION_POOL_SERVICE, DIRECTORY, SKIP_FILES_WITH_SUFFIX).contains(property);
+ }
+
+ @Override
+ protected Scope getStateScope(PropertyContext context) {
+ return CLUSTER;
+ }
+
+ @Override
+ protected RecordSchema getRecordSchema() {
+ return SmbListableEntity.getRecordSchema();
+ }
+
+ @Override
+ protected Integer countUnfilteredListing(ProcessContext context) throws IOException {
+ try (Stream<SmbListableEntity> listing = performListing(context)) {
+ return Long.valueOf(listing.count()).intValue();
+ } catch (IOException e) {
+ throw e;
+ } catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+
+ @Override
+ protected String getListingContainerName(ProcessContext context) {
+ return String.format("Samba Directory [%s]", getPath(context));
Review Comment:
`Samba` is a Linux-specific implementation of the SMB protocol. This should be adjusted:
```suggestion
return String.format("Remote Directory [%s]", getPath(context));
```
##########
nifi-nar-bundles/nifi-smb-bundle/pom.xml:
##########
@@ -26,7 +26,36 @@
<packaging>pom</packaging>
<modules>
+ <module>nifi-smb-connection-pool-api</module>
+ <module>nifi-smb-connection-pool-api-nar</module>
+ <module>nifi-smb-connection-pool</module>
+ <module>nifi-smb-connection-pool-nar</module>
<module>nifi-smb-processors</module>
<module>nifi-smb-nar</module>
</modules>
+
+ <dependencyManagement>
+ <dependencies>
+ <dependency>
+ <groupId>com.hierynomus</groupId>
+ <artifactId>smbj</artifactId>
+ <version>0.11.5</version>
+ </dependency>
+ <dependency>
+ <groupId>net.engio</groupId>
+ <artifactId>mbassador</artifactId>
+ <version>1.3.0</version>
Review Comment:
The latest version appears to be 1.3.2, is there a reason for not using that version?
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+
+import java.io.IOException;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbConnectionPoolService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file in the filesystem was modified as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum file age in milliseconds")
+ .name("min-age")
+ .description(
+ "Any file younger the the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(INTEGER_VALIDATOR)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5000")
+ .build();
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-connection-pool-service")
+ .displayName("SMB Connection Pool Service")
+ .description("Specifies the SMB Connection Pool to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbConnectionPoolService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("skip-files-with-suffix")
+ .displayName("File name suffix filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final String FILE_MODIFY_DATE_ATTR_FORMAT = "yyyy-MM-dd'T'HH:mm:ssZ";
+ private static final DateFormat formatter = new SimpleDateFormat(FILE_MODIFY_DATE_ATTR_FORMAT, Locale.US);
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ AbstractListProcessor.RECORD_WRITER,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET,
+ SMB_LISTING_STRATEGY,
+ SMB_CONNECTION_POOL_SERVICE,
+ DIRECTORY,
+ MINIMUM_AGE,
+ SKIP_FILES_WITH_SUFFIX
+ ));
+
+ NiFiSmbClientFactory smbClientFactory = new NiFiSmbClientFactory();
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Map<String, String> createAttributes(SmbListableEntity entity, ProcessContext context) {
+ final Map<String, String> attributes = new TreeMap<>();
+ attributes.put("filename", entity.getName());
+ attributes.put("path", entity.getPath());
+ attributes.put("absolute.path", getPath(context) + entity.getPathWithName());
+ attributes.put("identifier", entity.getIdentifier());
+ attributes.put("timestamp", formatter.format(new Date(entity.getTimestamp())));
+ attributes.put("size", String.valueOf(entity.getSize()));
+ return unmodifiableMap(attributes);
+ }
+
+ @Override
+ protected String getPath(ProcessContext context) {
+ final SmbConnectionPoolService connectionPoolService =
+ context.getProperty(SMB_CONNECTION_POOL_SERVICE).asControllerService(SmbConnectionPoolService.class);
+ final String hostname = connectionPoolService.getHostname();
+ final String port = String.valueOf(connectionPoolService.getPort());
+ final String share = connectionPoolService.getShareName();
+ final String directory = getDirectory(context);
+ return String.format("smb://%s:%s/%s:\\%s", hostname, port, share, directory.isEmpty() ? "" : directory + "\\");
+ }
+
+ @Override
+ protected List<SmbListableEntity> performListing(ProcessContext context, Long minimumTimestampOrNull,
+ ListingMode listingMode) throws IOException {
+
+ final Integer minimumAge = context.getProperty(MINIMUM_AGE).asInteger();
+ final String suffixOrNull = context.getProperty(SKIP_FILES_WITH_SUFFIX).getValue();
+ final Predicate<SmbListableEntity> fileFilter =
+ createFileFilter(minimumAge, minimumTimestampOrNull, suffixOrNull);
+ try (Stream<SmbListableEntity> listing = performListing(context)) {
+ final Iterator<SmbListableEntity> iterator = listing.iterator();
+ final List<SmbListableEntity> result = new LinkedList<>();
+ while (iterator.hasNext()) {
+ if (!isExecutionScheduled(listingMode)) {
+ return emptyList();
+ }
+ final SmbListableEntity entity = iterator.next();
+ if (fileFilter.test(entity)) {
+ result.add(entity);
+ }
+ }
+ return result;
+ } catch (IOException o) {
+ throw o;
+ } catch (Exception e) {
+ throw new IOException(e);
+ }
+ }
+
+ @Override
+ protected boolean isListingResetNecessary(PropertyDescriptor property) {
+ return asList(SMB_CONNECTION_POOL_SERVICE, DIRECTORY, SKIP_FILES_WITH_SUFFIX).contains(property);
+ }
+
+ @Override
+ protected Scope getStateScope(PropertyContext context) {
+ return CLUSTER;
+ }
+
+ @Override
+ protected RecordSchema getRecordSchema() {
+ return SmbListableEntity.getRecordSchema();
+ }
+
+ @Override
+ protected Integer countUnfilteredListing(ProcessContext context) throws IOException {
+ try (Stream<SmbListableEntity> listing = performListing(context)) {
+ return Long.valueOf(listing.count()).intValue();
+ } catch (IOException e) {
+ throw e;
+ } catch (Exception e) {
+ throw new IOException(e);
+ }
Review Comment:
See other comments on streamlining and adding a message.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjConnectionPoolService.java:
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.util.StringUtils;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjConnectionPoolService extends AbstractControllerService implements SmbConnectionPoolService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host to which files should be written.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be written. This is the \"first folder\"" +
+ "after the hostname: \\\\hostname\\[share]\\dir1\\dir2")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication. If no username is set then anonymous authentication is attempted.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication. Required if Username is set.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(INTEGER_VALIDATOR)
Review Comment:
This should be changed to use the Port Validator.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+
+import java.io.IOException;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbConnectionPoolService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file in the filesystem was modified as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum file age in milliseconds")
+ .name("min-age")
+ .description(
+ "Any file younger the the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(INTEGER_VALIDATOR)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5000")
+ .build();
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-connection-pool-service")
+ .displayName("SMB Connection Pool Service")
+ .description("Specifies the SMB Connection Pool to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbConnectionPoolService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("skip-files-with-suffix")
+ .displayName("File name suffix filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final String FILE_MODIFY_DATE_ATTR_FORMAT = "yyyy-MM-dd'T'HH:mm:ssZ";
+ private static final DateFormat formatter = new SimpleDateFormat(FILE_MODIFY_DATE_ATTR_FORMAT, Locale.US);
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ AbstractListProcessor.RECORD_WRITER,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET,
+ SMB_LISTING_STRATEGY,
+ SMB_CONNECTION_POOL_SERVICE,
+ DIRECTORY,
+ MINIMUM_AGE,
+ SKIP_FILES_WITH_SUFFIX
+ ));
+
+ NiFiSmbClientFactory smbClientFactory = new NiFiSmbClientFactory();
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Map<String, String> createAttributes(SmbListableEntity entity, ProcessContext context) {
+ final Map<String, String> attributes = new TreeMap<>();
+ attributes.put("filename", entity.getName());
+ attributes.put("path", entity.getPath());
+ attributes.put("absolute.path", getPath(context) + entity.getPathWithName());
+ attributes.put("identifier", entity.getIdentifier());
+ attributes.put("timestamp", formatter.format(new Date(entity.getTimestamp())));
+ attributes.put("size", String.valueOf(entity.getSize()));
+ return unmodifiableMap(attributes);
+ }
+
+ @Override
+ protected String getPath(ProcessContext context) {
+ final SmbConnectionPoolService connectionPoolService =
+ context.getProperty(SMB_CONNECTION_POOL_SERVICE).asControllerService(SmbConnectionPoolService.class);
+ final String hostname = connectionPoolService.getHostname();
+ final String port = String.valueOf(connectionPoolService.getPort());
+ final String share = connectionPoolService.getShareName();
+ final String directory = getDirectory(context);
+ return String.format("smb://%s:%s/%s:\\%s", hostname, port, share, directory.isEmpty() ? "" : directory + "\\");
+ }
+
+ @Override
+ protected List<SmbListableEntity> performListing(ProcessContext context, Long minimumTimestampOrNull,
+ ListingMode listingMode) throws IOException {
+
+ final Integer minimumAge = context.getProperty(MINIMUM_AGE).asInteger();
+ final String suffixOrNull = context.getProperty(SKIP_FILES_WITH_SUFFIX).getValue();
+ final Predicate<SmbListableEntity> fileFilter =
+ createFileFilter(minimumAge, minimumTimestampOrNull, suffixOrNull);
+ try (Stream<SmbListableEntity> listing = performListing(context)) {
+ final Iterator<SmbListableEntity> iterator = listing.iterator();
+ final List<SmbListableEntity> result = new LinkedList<>();
+ while (iterator.hasNext()) {
+ if (!isExecutionScheduled(listingMode)) {
+ return emptyList();
+ }
+ final SmbListableEntity entity = iterator.next();
+ if (fileFilter.test(entity)) {
+ result.add(entity);
+ }
+ }
+ return result;
+ } catch (IOException o) {
+ throw o;
+ } catch (Exception e) {
+ throw new IOException(e);
+ }
Review Comment:
This could be collapsed into a single catch block, and it would be helpful to include an exception message.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjConnectionPoolService.java:
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.util.StringUtils;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjConnectionPoolService extends AbstractControllerService implements SmbConnectionPoolService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host to which files should be written.")
+ .required(false)
+ .expressionLanguageSupported(FLOWFILE_ATTRIBUTES)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be written. This is the \"first folder\"" +
+ "after the hostname: \\\\hostname\\[share]\\dir1\\dir2")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication. If no username is set then anonymous authentication is attempted.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication. Required if Username is set.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(INTEGER_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(Arrays.asList(
+ HOSTNAME,
+ SHARE,
+ DOMAIN,
+ USERNAME,
+ PASSWORD,
+ PORT
+ ));
+
+ public SMBClient getSmbClient() {
+ return smbClient;
+ }
+
+ public AuthenticationContext getAuthenticationContext() {
+ return authenticationContext;
+ }
+
+ private final SMBClient smbClient = new SMBClient();
Review Comment:
The `SMBClient` is closeable`, so it should be instantiated in `onEnabled` and closed in a method annotated with `OnDisabled`.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+
+import java.io.IOException;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbConnectionPoolService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file in the filesystem was modified as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum file age in milliseconds")
+ .name("min-age")
+ .description(
+ "Any file younger the the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(INTEGER_VALIDATOR)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5000")
+ .build();
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-connection-pool-service")
+ .displayName("SMB Connection Pool Service")
+ .description("Specifies the SMB Connection Pool to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbConnectionPoolService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("skip-files-with-suffix")
+ .displayName("File name suffix filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final String FILE_MODIFY_DATE_ATTR_FORMAT = "yyyy-MM-dd'T'HH:mm:ssZ";
+ private static final DateFormat formatter = new SimpleDateFormat(FILE_MODIFY_DATE_ATTR_FORMAT, Locale.US);
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ AbstractListProcessor.RECORD_WRITER,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET,
+ SMB_LISTING_STRATEGY,
+ SMB_CONNECTION_POOL_SERVICE,
+ DIRECTORY,
+ MINIMUM_AGE,
+ SKIP_FILES_WITH_SUFFIX
+ ));
+
+ NiFiSmbClientFactory smbClientFactory = new NiFiSmbClientFactory();
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Map<String, String> createAttributes(SmbListableEntity entity, ProcessContext context) {
+ final Map<String, String> attributes = new TreeMap<>();
+ attributes.put("filename", entity.getName());
+ attributes.put("path", entity.getPath());
+ attributes.put("absolute.path", getPath(context) + entity.getPathWithName());
+ attributes.put("identifier", entity.getIdentifier());
+ attributes.put("timestamp", formatter.format(new Date(entity.getTimestamp())));
+ attributes.put("size", String.valueOf(entity.getSize()));
+ return unmodifiableMap(attributes);
+ }
+
+ @Override
+ protected String getPath(ProcessContext context) {
+ final SmbConnectionPoolService connectionPoolService =
+ context.getProperty(SMB_CONNECTION_POOL_SERVICE).asControllerService(SmbConnectionPoolService.class);
+ final String hostname = connectionPoolService.getHostname();
+ final String port = String.valueOf(connectionPoolService.getPort());
+ final String share = connectionPoolService.getShareName();
+ final String directory = getDirectory(context);
+ return String.format("smb://%s:%s/%s:\\%s", hostname, port, share, directory.isEmpty() ? "" : directory + "\\");
Review Comment:
As mentioned on the interface comments, the leading portion of this URI could be provided through a single service method, then this processor method could just append the directory.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR;
+
+import java.io.IOException;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbConnectionPoolService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file in the filesystem was modified as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum file age in milliseconds")
+ .name("min-age")
+ .description(
+ "Any file younger the the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(INTEGER_VALIDATOR)
+ .addValidator(NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5000")
+ .build();
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-connection-pool-service")
+ .displayName("SMB Connection Pool Service")
+ .description("Specifies the SMB Connection Pool to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbConnectionPoolService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("skip-files-with-suffix")
+ .displayName("File name suffix filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final String FILE_MODIFY_DATE_ATTR_FORMAT = "yyyy-MM-dd'T'HH:mm:ssZ";
+ private static final DateFormat formatter = new SimpleDateFormat(FILE_MODIFY_DATE_ATTR_FORMAT, Locale.US);
Review Comment:
`SimpleDateFormat` is not thread safe. Using the Java 8 `DateTimeFormatter` provides a better option.
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjConnectionPoolService.java:
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.util.StringUtils;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjConnectionPoolService extends AbstractControllerService implements SmbConnectionPoolService {
Review Comment:
It would be helpful to provide a property for configuring the Timeout on SMBClient connections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] kulikg commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
kulikg commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r922251851
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-connection-pool/src/main/java/org/apache/nifi/services/smb/SmbjSessionProviderService.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static org.apache.nifi.processor.util.StandardValidators.INTEGER_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.transport.tcp.async.AsyncDirectTcpTransportFactory;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"microsoft", "samba"})
+@CapabilityDescription("Provides connection pool for ListSmb processor. ")
+public class SmbjSessionProviderService extends AbstractControllerService implements SmbSessionProviderService {
Review Comment:
Good point. I think the api shouldn't depend on smbj and this should be fixed. As you suggested to extract the nifi smb client interface I think the best would be to lift that interface right into the api and move the implementation into the controller service module.
This way both the api and the processor would be independent from smbj so that other java driver could be easily added, also I could get rid of the NiFiSmbClientFactory and move that functionality into the controller service and mock it nicely using mock framework in the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r928320217
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"samba, smb, cifs, files", "list"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Lists concrete files shared via SMB protocol. " +
+ "Each listed file may result in one flowfile, the metadata being written as flowfile attributes. " +
+ "Or - in case the 'Record Writer' property is set - the entire result is written as records to a single flowfile. " +
+ "This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the " +
+ "previous node left off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:\\DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder from which to list files. This is the remaining relative path " +
+ "after the share: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + "to add subdirectories. The given path on the remote file share must exist. "
+ + "This can be checked using verification. You may mix Windows and Linux-style "
+ + "directory separators.")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description("The minimum age that a file must be in order to be listed; any file younger than this "
+ + "amount of time will be ignored.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Age")
+ .name("max-file-age")
+ .description("Any file older than the given value will be omitted. ")
+ .required(false)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller than the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size")
+ .name("max-file-size")
+ .description("Any file larger than the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CLIENT_PROVIDER_SERVICE = new Builder()
+ .name("smb-client-provider-service")
+ .displayName("SMB Client Provider Service")
+ .description("Specifies the SMB client provider to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbClientProviderService.class)
+ .build();
+
+ public static final PropertyDescriptor FILE_NAME_SUFFIX_FILTER = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File Name Suffix Filter")
+ .description("Files ending with the given suffix will be omitted. Can be used to make sure that files "
+ + "that are still uploading are not listed multiple times, by having those files have a suffix "
+ + "and remove the suffix once the upload finishes. This is highly recommended when using "
+ + "'Tracking Entities' or 'Tracking Timestamps' listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ SMB_LISTING_STRATEGY,
+ SMB_CLIENT_PROVIDER_SERVICE,
+ DIRECTORY,
+ AbstractListProcessor.RECORD_WRITER,
+ FILE_NAME_SUFFIX_FILTER,
+ MINIMUM_AGE,
+ MAXIMUM_AGE,
+ MINIMUM_SIZE,
+ MAXIMUM_SIZE,
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET
+ ));
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Map<String, String> createAttributes(SmbListableEntity entity, ProcessContext context) {
+ final Map<String, String> attributes = new TreeMap<>();
+ attributes.put("filename", entity.getName());
+ attributes.put("shortname", entity.getShortName());
+ attributes.put("path", entity.getPath());
+ attributes.put("absolute.path", getPath(context) + entity.getPathWithName());
+ attributes.put("identifier", entity.getIdentifier());
+ attributes.put("timestamp", formatTimeStamp(entity.getTimestamp()));
+ attributes.put("creationTime", formatTimeStamp(entity.getCreationTime()));
+ attributes.put("lastAccessTime", formatTimeStamp(entity.getLastAccessTime()));
+ attributes.put("changeTime", formatTimeStamp(entity.getChangeTime()));
+ attributes.put("size", String.valueOf(entity.getSize()));
+ attributes.put("allocationSize", String.valueOf(entity.getAllocationSize()));
+ return unmodifiableMap(attributes);
+ }
+
+ @Override
+ protected String getPath(ProcessContext context) {
+ final SmbClientProviderService clientProviderService =
+ context.getProperty(SMB_CLIENT_PROVIDER_SERVICE).asControllerService(SmbClientProviderService.class);
+ final URI serviceLocation = clientProviderService.getServiceLocation();
+ final String directory = getDirectory(context);
+ return String.format("%s:\\%s", serviceLocation.toString(), directory.isEmpty() ? "" : directory + "\\");
Review Comment:
The `:` after the share name seems to be a typo. There should not be a colon there.
I would suggest using forward slashes (`/`) in SMB URIs instead of backslash (`\`). Mixing them looks to me a bit strange and forward slash seems to be the standard (though it is a draft only):
https://datatracker.ietf.org/doc/html/draft-crhertel-smb-url-12#section-7
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"samba, smb, cifs, files", "list"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Lists concrete files shared via SMB protocol. " +
+ "Each listed file may result in one flowfile, the metadata being written as flowfile attributes. " +
+ "Or - in case the 'Record Writer' property is set - the entire result is written as records to a single flowfile. " +
+ "This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the " +
+ "previous node left off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:\\DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder from which to list files. This is the remaining relative path " +
+ "after the share: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + "to add subdirectories. The given path on the remote file share must exist. "
+ + "This can be checked using verification. You may mix Windows and Linux-style "
+ + "directory separators.")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description("The minimum age that a file must be in order to be listed; any file younger than this "
+ + "amount of time will be ignored.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Age")
+ .name("max-file-age")
+ .description("Any file older than the given value will be omitted. ")
+ .required(false)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller than the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size")
+ .name("max-file-size")
+ .description("Any file larger than the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CLIENT_PROVIDER_SERVICE = new Builder()
+ .name("smb-client-provider-service")
+ .displayName("SMB Client Provider Service")
+ .description("Specifies the SMB client provider to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbClientProviderService.class)
+ .build();
+
+ public static final PropertyDescriptor FILE_NAME_SUFFIX_FILTER = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File Name Suffix Filter")
+ .description("Files ending with the given suffix will be omitted. Can be used to make sure that files "
+ + "that are still uploading are not listed multiple times, by having those files have a suffix "
+ + "and remove the suffix once the upload finishes. This is highly recommended when using "
+ + "'Tracking Entities' or 'Tracking Timestamps' listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ SMB_LISTING_STRATEGY,
+ SMB_CLIENT_PROVIDER_SERVICE,
+ DIRECTORY,
+ AbstractListProcessor.RECORD_WRITER,
+ FILE_NAME_SUFFIX_FILTER,
+ MINIMUM_AGE,
+ MAXIMUM_AGE,
+ MINIMUM_SIZE,
+ MAXIMUM_SIZE,
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET
+ ));
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Map<String, String> createAttributes(SmbListableEntity entity, ProcessContext context) {
+ final Map<String, String> attributes = new TreeMap<>();
+ attributes.put("filename", entity.getName());
+ attributes.put("shortname", entity.getShortName());
+ attributes.put("path", entity.getPath());
+ attributes.put("absolute.path", getPath(context) + entity.getPathWithName());
Review Comment:
If I specify a directory in the `Input Directory` property, then the directory appears twice in the SMB URL: `smb://localhost:445/public:\dir1\dir1\dir2\file3`
Suggesting forward slashes here too. Also in `identifier` and `path`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] tpalfy commented on pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
tpalfy commented on PR #6192:
URL: https://github.com/apache/nifi/pull/6192#issuecomment-1206651747
LGTM
Thanks for your work @kulikg!
Pushed to main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
URL: https://github.com/apache/nifi/pull/6192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] kulikg commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
kulikg commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r926544342
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/ListSmb.java:
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.time.format.DateTimeFormatter.ISO_DATE_TIME;
+import static java.util.Arrays.asList;
+import static java.util.Collections.emptyList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static org.apache.nifi.processor.util.StandardValidators.DATA_SIZE_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import java.io.IOException;
+import java.net.URI;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+import org.apache.nifi.processor.util.list.ListedEntityTracker;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.services.smb.SmbClientService;
+import org.apache.nifi.services.smb.SmbListableEntity;
+import org.apache.nifi.services.smb.SmbClientProviderService;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@Tags({"microsoft", "storage", "samba"})
+@SeeAlso({PutSmbFile.class, GetSmbFile.class})
+@CapabilityDescription("Retrieves a listing of files shared via SMB protocol. For each file that is listed, " +
+ "creates a FlowFile that represents the file. This Processor is designed to run on Primary Node only in " +
+ "a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left " +
+ "off without duplicating all of the data.")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "shortname", description = "The short name of the file that was read from filesystem."),
+ @WritesAttribute(attribute = "path", description =
+ "The path is set to the relative path of the file's directory "
+ + "on filesystem compared to the Share and Input Directory properties and the configured host "
+ + "and port inherited from the configured connection pool controller service. For example, for "
+ + "a given remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listed from "
+ + "smb://HOSTNAME:PORT/SHARE:DIRECTORY\\sub\\folder\\file then the path attribute will be set to \"sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "absolute.path", description =
+ "The absolute.path is set to the absolute path of the file's directory on the remote location. For example, "
+ + "given a remote location smb://HOSTNAME:PORT/SHARE:\\DIRECTORY, and a file is being listen from "
+ + "SHARE:\\DIRECTORY\\sub\\folder\\file then the absolute.path attribute will be set to "
+ + "\"SHARE:\\DIRECTORY\\sub\\folder\\file\"."),
+ @WritesAttribute(attribute = "identifier", description =
+ "The identifier of the file. This equals to the path attribute so two files with the same relative path "
+ + "coming from different file shares considered to be identical."),
+ @WritesAttribute(attribute = "timestamp", description =
+ "The timestamp of when the file's content in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "createTime", description =
+ "The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "lastAccessTime", description =
+ "The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "changeTime", description =
+ "The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ssZ'"),
+ @WritesAttribute(attribute = "size", description = "The number of bytes in the source file"),
+ @WritesAttribute(attribute = "allocationSize", description = "The number of bytes allocated for the file on the server"),
+})
+@Stateful(scopes = {Scope.CLUSTER}, description =
+ "After performing a listing of files, the state of the previous listing can be stored in order to list files "
+ + "continuously without duplication."
+)
+public class ListSmb extends AbstractListProcessor<SmbListableEntity> {
+
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .displayName("Input Directory")
+ .name("directory")
+ .description("The network folder to which files should be written. This is the remaining relative " +
+ "after the hostname: smb://HOSTNAME:PORT/SHARE/[DIRECTORY]\\sub\\directories. It is also possible "
+ + " to add subdirectories using this property. The given path on the remote file share must exists. "
+ + "The existence of the remote folder can be checked using verification. You may mix different "
+ + "directory separators in this property. If so NiFi will unify all of them and will use windows's"
+ + "directory separator: '\\' ")
+ .required(false)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_AGE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Age")
+ .name("min-file-age")
+ .description(
+ "Any file younger then the given value will be omitted. Ideally this value should be greater then"
+ + "the amount of time needed to perform a list.")
+ .required(true)
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 secs")
+ .build();
+
+ public static final PropertyDescriptor MINIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Minimum File Size")
+ .name("min-file-size")
+ .description("Any file smaller then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor MAXIMUM_SIZE = new PropertyDescriptor.Builder()
+ .displayName("Maximum File Size")
+ .name("max-file-size")
+ .description("Any file bigger then the given value will be omitted.")
+ .required(false)
+ .addValidator(DATA_SIZE_VALIDATOR)
+ .build();
+
+
+ public static final PropertyDescriptor SMB_LISTING_STRATEGY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(LISTING_STRATEGY)
+ .allowableValues(BY_ENTITIES, NO_TRACKING, BY_TIMESTAMPS)
+ .build();
+
+ public static final PropertyDescriptor SMB_CONNECTION_POOL_SERVICE = new Builder()
+ .name("smb-client-provider-service")
+ .displayName("SMB Client Provider Service")
+ .description("Specifies the SMB client provider to use for creating SMB connections.")
+ .required(true)
+ .identifiesControllerService(SmbClientProviderService.class)
+ .build();
+
+ public static final PropertyDescriptor SKIP_FILES_WITH_SUFFIX = new Builder()
+ .name("file-name-suffix-filter")
+ .displayName("File Name Suffix Filter")
+ .description("Files ends with the given suffix will be omitted. This is handy when writing large data into "
+ + "temporary files and then moved to a final one. Please be advised that writing data into files "
+ + "first is highly recommended when using Entity Tracking or Timestamp based listing strategies.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .addValidator(new MustNotContainDirectorySeparatorsValidator())
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTIES = unmodifiableList(asList(
+ SMB_LISTING_STRATEGY,
+ SMB_CONNECTION_POOL_SERVICE,
+ DIRECTORY,
+ AbstractListProcessor.RECORD_WRITER,
+ SKIP_FILES_WITH_SUFFIX,
+ MINIMUM_AGE,
+ MINIMUM_SIZE,
+ MAXIMUM_SIZE,
+ AbstractListProcessor.TARGET_SYSTEM_TIMESTAMP_PRECISION,
+ ListedEntityTracker.TRACKING_STATE_CACHE,
+ ListedEntityTracker.TRACKING_TIME_WINDOW,
+ ListedEntityTracker.INITIAL_LISTING_TARGET
+ ));
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @Override
+ protected Map<String, String> createAttributes(SmbListableEntity entity, ProcessContext context) {
+ final Map<String, String> attributes = new TreeMap<>();
+ attributes.put("filename", entity.getName());
+ attributes.put("shortname", entity.getShortName());
+ attributes.put("path", entity.getPath());
+ attributes.put("absolute.path", getPath(context) + entity.getPathWithName());
+ attributes.put("identifier", entity.getIdentifier());
+ attributes.put("timestamp",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getTimestamp(), 0, ZoneOffset.UTC)));
+ attributes.put("creationTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getCreationTime(), 0, ZoneOffset.UTC)));
+ attributes.put("lastAccessedTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getLastAccessTime(), 0, ZoneOffset.UTC)));
+ attributes.put("changeTime",
+ ISO_DATE_TIME.format(LocalDateTime.ofEpochSecond(entity.getChangeTime(), 0, ZoneOffset.UTC)));
+ attributes.put("size", String.valueOf(entity.getSize()));
+ attributes.put("allocationSize", String.valueOf(entity.getAllocationSize()));
+ return unmodifiableMap(attributes);
+ }
+
+ @Override
+ protected String getPath(ProcessContext context) {
+ final SmbClientProviderService connectionPoolService =
+ context.getProperty(SMB_CONNECTION_POOL_SERVICE).asControllerService(SmbClientProviderService.class);
+ final URI serviceLocation = connectionPoolService.getServiceLocation();
+ final String directory = getDirectory(context);
+ return String.format("%s:\\%s", serviceLocation.toString(), directory.isEmpty() ? "" : directory + "\\");
+ }
+
+ @Override
+ protected List<SmbListableEntity> performListing(ProcessContext context, Long minimumTimestampOrNull,
+ ListingMode listingMode) throws IOException {
+
+ final Predicate<SmbListableEntity> fileFilter =
+ createFileFilter(context, minimumTimestampOrNull);
+
+ try (Stream<SmbListableEntity> listing = performListing(context)) {
+ final Iterator<SmbListableEntity> iterator = listing.iterator();
+ final List<SmbListableEntity> result = new LinkedList<>();
+ while (iterator.hasNext()) {
+ if (!isExecutionScheduled(listingMode)) {
+ return emptyList();
+ }
+ final SmbListableEntity entity = iterator.next();
+ if (fileFilter.test(entity)) {
+ result.add(entity);
+ }
+ }
+ return result;
Review Comment:
Please note that a processor can be stopped during listing.
Do you have any suggestion how I could do that using the stream api?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r922298770
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/NiFiSmbClient.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.stream.StreamSupport.stream;
+
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msfscc.fileinformation.FileIdBothDirectoryInformation;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMBApiException;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.Directory;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.stream.Stream;
+
+public class NiFiSmbClient {
+
+ private static final List<String> SPECIAL_DIRECTORIES = asList(".", "..");
Review Comment:
Thanks for the reply, that's a good point. Leaving the variable name as `SPECIAL_DIRECTORIES` sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] kulikg commented on pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
kulikg commented on PR #6192:
URL: https://github.com/apache/nifi/pull/6192#issuecomment-1185218734
> @kulikg - Did you have the intention of refactoring the existing GetSmb and PutSmb processors to utilise the client? Or do you think this is a body of work for later?
It would be really nice to refactor them but I'm not planning to do so in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] tpalfy commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
tpalfy commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r934871047
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-nar/pom.xml:
##########
@@ -35,5 +35,16 @@
<artifactId>nifi-smb-processors</artifactId>
<version>1.17.0-SNAPSHOT</version>
</dependency>
+ <!--dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smb-smbj-client</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </dependency-->
Review Comment:
I think we don't want to merge it this way. Can we remove it instead of just commenting out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r928315803
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-client/src/main/java/org/apache/nifi/services/smb/SmbjClientProviderService.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.smb;
+
+import static java.util.Arrays.asList;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.PORT_VALIDATOR;
+import static org.apache.nifi.processor.util.StandardValidators.TIME_PERIOD_VALIDATOR;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.SmbConfig;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.Share;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+
+@Tags({"samba, smb, cifs, files"})
+@CapabilityDescription("Provides access to SMB Sessions with shared authentication credentials.")
+public class SmbjClientProviderService extends AbstractControllerService implements SmbClientProviderService {
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .displayName("Hostname")
+ .name("hostname")
+ .description("The network host of the SMB file server.")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .displayName("Domain")
+ .name("domain")
+ .description(
+ "The domain used for authentication. Optional, in most cases username and password is sufficient.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .displayName("Username")
+ .name("username")
+ .description(
+ "The username used for authentication.")
+ .required(false)
+ .defaultValue("Guest")
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .displayName("Password")
+ .name("password")
+ .description("The password used for authentication.")
+ .required(false)
+ .addValidator(NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+ .displayName("Port")
+ .name("port")
+ .description("Port to use for connection.")
+ .required(true)
+ .addValidator(PORT_VALIDATOR)
+ .defaultValue("445")
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .displayName("Share")
+ .name("share")
+ .description("The network share to which files should be listed from. This is the \"first folder\"" +
+ "after the hostname: smb://hostname:port\\[share]\\dir1\\dir2")
+ .required(true)
+ .addValidator(NON_BLANK_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TIMEOUT = new PropertyDescriptor.Builder()
+ .displayName("Timeout")
+ .name("timeout")
+ .description("Timeout for read and write operations.")
+ .required(true)
+ .defaultValue("5 secs")
+ .addValidator(TIME_PERIOD_VALIDATOR)
+ .build();
+ private static final List<PropertyDescriptor> PROPERTIES = Collections
+ .unmodifiableList(asList(
+ HOSTNAME,
+ PORT,
+ SHARE,
+ USERNAME,
+ PASSWORD,
+ DOMAIN,
+ TIMEOUT
+ ));
+ private SMBClient smbClient;
+ private AuthenticationContext authenticationContext;
+ private String hostname;
+ private int port;
+ private String shareName;
+
+ @Override
+ public SmbClientService getClient() {
+ try {
+ final Connection connection = smbClient.connect(hostname, port);
+ final Session session = connection.authenticate(authenticationContext);
Review Comment:
@kulikg It seems the connection can become stale and unusable under some circumstances. In these cases, `connection.isConnected()` still returns `true` (that's why `SMBClient` does not create a new one) but `connection.authenticate(...)` fails.
I ran into the following cases:
1. Using SMB server running in a Docker container locally. After 60 seconds (even when runnig in every 2 seconds, so no inactivity):
```
com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
at com.hierynomus.smbj.connection.SMBSessionBuilder.establish(SMBSessionBuilder.java:114)
at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:202)
at org.apache.nifi.services.smb.SmbjClientProviderService.getClient(SmbjClientProviderService.java:126)
```
2. Using a Windows share. After some idle / inactive time:
```
com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: java.net.SocketException: Broken pipe (Write failed)
at com.hierynomus.smbj.connection.SMBSessionBuilder.establish(SMBSessionBuilder.java:114)
at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:202)
at org.apache.nifi.services.smb.SmbjClientProviderService.getClient(SmbjClientProviderService.java:126)
```
It seems to be a bug in the smbj library or some extra config may help.
As a workaround, the exceptions above could be caught and the connection could be caused forcefully and then `SMBClient.connect()` would create a new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6192: NIFI-10212 added ListSmb processor and SmbConnectionPoolService
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on code in PR #6192:
URL: https://github.com/apache/nifi/pull/6192#discussion_r925534235
##########
nifi-nar-bundles/nifi-smb-bundle/nifi-smb-nar/pom.xml:
##########
@@ -35,5 +35,16 @@
<artifactId>nifi-smb-processors</artifactId>
<version>1.17.0-SNAPSHOT</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-smbj-client</artifactId>
+ <version>1.17.0-SNAPSHOT</version>
+ </dependency>
Review Comment:
The processor nar should not depend on a controller service implementation. As far as I can see, it is not used and can just be deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org