You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2020/03/30 09:59:04 UTC
[GitHub] [nifi] si-sun opened a new pull request #4169: Adding GetSmbFile
and PutSmbFile processors #2
si-sun opened a new pull request #4169: Adding GetSmbFile and PutSmbFile processors #2
URL: https://github.com/apache/nifi/pull/4169
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
#### Description of PR
Added functionality to read and write from samba shares with GetSmbFile and PutSmbFile processors. The advantage is to not have to manage cifs mounts on the server/nodes. Also the processors supports windows style share locks on the files during read and write.
It's using [SMBJ client library](https://github.com/hierynomus/smbj) for the samba connection.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-XXXX** where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [X] Has your PR been rebased against the latest commit within the target branch (typically `master`)?
- [X] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._
### For code changes:
- [X] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder?
- [X] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and JDK 11?
- [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`?
- [X] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] joewitt commented on issue #4169: Adding GetSmbFile and
PutSmbFile processors #2
Posted by GitBox <gi...@apache.org>.
joewitt commented on issue #4169: Adding GetSmbFile and PutSmbFile processors #2
URL: https://github.com/apache/nifi/pull/4169#issuecomment-610450697
@si-sun What JIRA is this associated with? We need the commit heading to reference the NIFI jIRA.
Please create a JIRA here https://issues.apache.org/jira/projects/NIFI/issues and update the commit message to reflect that. The PR itself looks good.
Only thing that jumped out is we might not include the nar in the default assembly due to size concerns but I'll check that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] joewitt commented on a change in pull request #4169:
NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
Posted by GitBox <gi...@apache.org>.
joewitt commented on a change in pull request #4169: NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
URL: https://github.com/apache/nifi/pull/4169#discussion_r405707790
##########
File path: nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/GetSmbFile.java
##########
@@ -0,0 +1,521 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.mserref.NtStatus;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msfscc.fileinformation.FileAllInformation;
+import com.hierynomus.msfscc.fileinformation.FileBasicInformation;
+import com.hierynomus.msfscc.fileinformation.FileIdBothDirectoryInformation;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMBApiException;
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.share.File;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.InputStream;
+
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.List;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.Collections;
+import java.util.ListIterator;
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.ArrayList;
+import java.util.Locale;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.regex.Pattern;
+
+@TriggerWhenEmpty
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"samba, smb, cifs, files, get"})
+@CapabilityDescription("Reads file from a samba network location to FlowFiles. " +
+ "Use this processor instead of a cifs mounts if share access control is important.")
+@SeeAlso({PutSmbFile.class})
+@WritesAttributes({
+ @WritesAttribute(attribute = "filename", description = "The filename is set to the name of the file on the network share"),
+ @WritesAttribute(attribute = "path", description = "The path is set to the relative path of the file's network share name. For example, "
+ + "if the input is set to \\\\hostname\\share\\tmp, files picked up from \\tmp will have the path attribute set to tmp"),
+ @WritesAttribute(attribute = "file.creationTime", description = "The date and time that the file was created. May not work on all file systems"),
+ @WritesAttribute(attribute = "file.lastModifiedTime", description = "The date and time that the file was last modified. May not work on all "
+ + "file systems"),
+ @WritesAttribute(attribute = "file.lastAccessTime", description = "The date and time that the file was last accessed. May not work on all "
+ + "file systems"),
+ @WritesAttribute(attribute = "absolute.path", description = "The full path from where a file was picked up. This includes "
+ + "the hostname and the share name")})
+public class GetSmbFile extends AbstractProcessor {
+ public static final String SHARE_ACCESS_NONE = "none";
+ public static final String SHARE_ACCESS_READ = "read";
+ public static final String SHARE_ACCESS_READDELETE = "read, delete";
+ public static final String SHARE_ACCESS_READWRITEDELETE = "read, write, delete";
+
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .name("Hostname")
+ .description("The network host to which files should be written.")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .name("Share")
+ .description("The network share to which files should be written.")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .name("Directory")
+ .description("The network folder to which files should be written. You may use expression language.")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .name("Domain")
+ .description("The domain use for authentication")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .name("Username")
+ .description("The username use for authentication")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .name("Password")
+ .description("The password use for authentication")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+
+ public static final PropertyDescriptor SHARE_ACCESS = new PropertyDescriptor.Builder()
+ .name("Share Access Strategy")
+ .description("Indicates which shared access are granted on the file during the read. " +
+ "None is the most restrictive, but the safest setting to prevent corruption.")
+ .required(true)
+ .defaultValue(SHARE_ACCESS_NONE)
+ .allowableValues(SHARE_ACCESS_NONE, SHARE_ACCESS_READ, SHARE_ACCESS_READDELETE, SHARE_ACCESS_READWRITEDELETE)
+ .build();
+ public static final PropertyDescriptor RECURSE = new PropertyDescriptor.Builder()
+ .name("Recurse Subdirectories")
+ .description("Indicates whether or not to pull files from subdirectories")
+ .required(true)
+ .allowableValues("true", "false")
+ .defaultValue("true")
+ .build();
+ public static final PropertyDescriptor KEEP_SOURCE_FILE = new PropertyDescriptor.Builder()
+ .name("Keep Source File")
+ .description("If true, the file is not deleted after it has been copied to the Content Repository; "
+ + "this causes the file to be picked up continually and is useful for testing purposes. "
+ + "If not keeping original NiFi will need write permissions on the directory it is pulling "
+ + "from otherwise it will ignore the file.")
+ .required(true)
+ .allowableValues("true", "false")
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor FILE_FILTER = new PropertyDescriptor.Builder()
+ .name("File Filter")
+ .description("Only files whose names match the given regular expression will be picked up")
+ .required(false)
+ .addValidator(StandardValidators.REGULAR_EXPRESSION_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PATH_FILTER = new PropertyDescriptor.Builder()
+ .name("Path Filter")
+ .description("When " + RECURSE.getName() + " is true, then only subdirectories whose path matches the given regular expression will be scanned")
+ .required(false)
+ .addValidator(StandardValidators.REGULAR_EXPRESSION_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IGNORE_HIDDEN_FILES = new PropertyDescriptor.Builder()
+ .name("Ignore Hidden Files")
+ .description("Indicates whether or not hidden files should be ignored")
+ .allowableValues("true", "false")
+ .defaultValue("true")
+ .required(true)
+ .build();
+ public static final PropertyDescriptor POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Polling Interval")
+ .description("Indicates how long to wait before performing a directory listing")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("0 sec")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .description("The maximum number of files to pull in each iteration")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+
+ public static final String FILE_CREATION_TIME_ATTRIBUTE = "file.creationTime";
+ public static final String FILE_LAST_MODIFY_TIME_ATTRIBUTE = "file.lastModifiedTime";
+ public static final String FILE_LAST_ACCESS_TIME_ATTRIBUTE = "file.lastAccessTime";
+
+ public static final String FILE_MODIFY_DATE_ATTR_FORMAT = "yyyy-MM-dd'T'HH:mm:ssZ";
+ final static DateFormat dateFormatter = new SimpleDateFormat(FILE_MODIFY_DATE_ATTR_FORMAT, Locale.US);
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success").description("All files are routed to success").build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+
+ private final BlockingQueue<String> fileQueue = new LinkedBlockingQueue<>();
+ private final Set<String> inProcess = new HashSet<>(); // guarded by queueLock
+ private final Set<String> recentlyProcessed = new HashSet<>(); // guarded by queueLock
+ private final Lock queueLock = new ReentrantLock();
+
+ private final Lock listingLock = new ReentrantLock();
+
+ private final AtomicLong queueLastUpdated = new AtomicLong(0L);
+
+ private SMBClient smbClient = null;
Review comment:
is this client thread safe? If not this processor should be set to triggerserially to avoid multiple threads operating at once.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] joewitt commented on issue #4169: NIFI-7338 Adding
GetSmbFile and PutSmbFile processors #2
Posted by GitBox <gi...@apache.org>.
joewitt commented on issue #4169: NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
URL: https://github.com/apache/nifi/pull/4169#issuecomment-611128345
Ok I've made it to the point that I can run GetSMbFile and pull files from a given directory and write those files back with PutSMBFile to another directory. Good.
1. It seems like 'directory' should be optional in the event we want to pull from the directory a given share is rooted at.
2. If username is set then password must be set. customValidate would be good to use here so the user knows this before attempting to run it.
3. There are no provenance events created for the SEND. There should be. See PutFile for an example.
4. The rate at which logs are spewed out is not helpful. The logs should be turned into debug at most. These are all like 'com.hierynomus.smbj.connection.Connection Successfully connected...bla' and 'com.hierynomus.smbj.session.Session ...' and 'com.hierynomus.smbj..transport.PacketReader ....bla'
5. The docs for the processor should be stronger and in particular the properties themselves.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] joewitt commented on a change in pull request #4169:
NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
Posted by GitBox <gi...@apache.org>.
joewitt commented on a change in pull request #4169: NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
URL: https://github.com/apache/nifi/pull/4169#discussion_r405707328
##########
File path: nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/PutSmbFile.java
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.smbj.share.File;
+import java.io.OutputStream;
+import java.util.EnumSet;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"samba, smb, cifs, files, put"})
+@CapabilityDescription("Writes the contents of a FlowFile to a samba network location. " +
+ "Use this processor instead of a cifs mounts if share access control is important.")
+@SeeAlso({GetSmbFile.class})
+@ReadsAttributes({@ReadsAttribute(attribute="filename", description="The filename to use when writing the FlowFile to the network folder.")})
+public class PutSmbFile extends AbstractProcessor {
+ public static final String SHARE_ACCESS_NONE = "none";
+ public static final String SHARE_ACCESS_READ = "read";
+ public static final String SHARE_ACCESS_READDELETE = "read, delete";
+ public static final String SHARE_ACCESS_READWRITEDELETE = "read, write, delete";
+
+ public static final String REPLACE_RESOLUTION = "replace";
+ public static final String IGNORE_RESOLUTION = "ignore";
+ public static final String FAIL_RESOLUTION = "fail";
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .name("Hostname")
+ .description("The network host to which files should be written.")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .name("Share")
+ .description("The network share to which files should be written.")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .name("Directory")
+ .description("The network folder to which files should be written. You may use expression language.")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .name("Domain")
+ .description("The domain use for authentication")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .name("Username")
+ .description("The username use for authentication")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .name("Password")
+ .description("The password use for authentication")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor CREATE_DIRS = new PropertyDescriptor.Builder()
+ .name("Create Missing Directories")
+ .description("If true, then missing destination directories will be created. If false, flowfiles are penalized and sent to failure.")
+ .required(true)
+ .allowableValues("true", "false")
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor SHARE_ACCESS = new PropertyDescriptor.Builder()
+ .name("Share Access Strategy")
+ .description("Indicates which shared access are granted on the file during the write. " +
+ "None is the most restrictive, but the safest setting to prevent corruption.")
+ .required(true)
+ .defaultValue(SHARE_ACCESS_NONE)
+ .allowableValues(SHARE_ACCESS_NONE, SHARE_ACCESS_READ, SHARE_ACCESS_READDELETE, SHARE_ACCESS_READWRITEDELETE)
+ .build();
+ public static final PropertyDescriptor CONFLICT_RESOLUTION = new PropertyDescriptor.Builder()
+ .name("Conflict Resolution Strategy")
+ .description("Indicates what should happen when a file with the same name already exists in the output directory")
+ .required(true)
+ .defaultValue(REPLACE_RESOLUTION)
+ .allowableValues(REPLACE_RESOLUTION, IGNORE_RESOLUTION, FAIL_RESOLUTION)
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .description("The maximum number of files to put in each iteration")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("100")
+ .build();
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Files that have been successfully written to the output network path are transferred to this relationship")
+ .build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Files that could not be written to the output network path for some reason are transferred to this relationship")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ private SMBClient smbClient = null;
Review comment:
is this client thread safe? If not this processor should set triggerSerially or this client should be protected
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] si-sun commented on issue #4169: NIFI-7338 Adding GetSmbFile
and PutSmbFile processors #2
Posted by GitBox <gi...@apache.org>.
si-sun commented on issue #4169: NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
URL: https://github.com/apache/nifi/pull/4169#issuecomment-611014872
alright added [NIFI-7338](https://issues.apache.org/jira/browse/NIFI-7338)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] joewitt commented on issue #4169: NIFI-7338 Adding
GetSmbFile and PutSmbFile processors #2
Posted by GitBox <gi...@apache.org>.
joewitt commented on issue #4169: NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
URL: https://github.com/apache/nifi/pull/4169#issuecomment-611017502
@si-sun thanks. The commit message should reflect the JIRA involved. So instead of it saying "Adding GetSmbFile and PutSmbFile processors" it should say "NIFI-7338 Adding GetSmbFile and PutSmbFile processors". You can see this in the PR notice text we provide and the contribution guide.
This is important because it causes the linkage of the commit to the JIRA to occur which greatly helps with release processes and tracking provenance of changes/etc..
I can do this if it gets merged but want to make sure you know this for future contributions or if you end up needing to add a commit to this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] si-sun commented on issue #4169: NIFI-7338 Adding GetSmbFile
and PutSmbFile processors #2
Posted by GitBox <gi...@apache.org>.
si-sun commented on issue #4169: NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
URL: https://github.com/apache/nifi/pull/4169#issuecomment-611425134
thanks @joewitt for the review and it's great to see some movement. I will look into the points and get back to you, might take a few days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] si-sun commented on a change in pull request #4169:
NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
Posted by GitBox <gi...@apache.org>.
si-sun commented on a change in pull request #4169: NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
URL: https://github.com/apache/nifi/pull/4169#discussion_r406063540
##########
File path: nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/PutSmbFile.java
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.smb;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import com.hierynomus.smbj.SMBClient;
+import com.hierynomus.smbj.connection.Connection;
+import com.hierynomus.smbj.auth.AuthenticationContext;
+import com.hierynomus.smbj.share.DiskShare;
+import com.hierynomus.smbj.session.Session;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.msdtyp.AccessMask;
+import com.hierynomus.msfscc.FileAttributes;
+import com.hierynomus.mssmb2.SMB2ShareAccess;
+import com.hierynomus.mssmb2.SMB2CreateDisposition;
+import com.hierynomus.mssmb2.SMB2CreateOptions;
+import com.hierynomus.smbj.share.File;
+import java.io.OutputStream;
+import java.util.EnumSet;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"samba, smb, cifs, files, put"})
+@CapabilityDescription("Writes the contents of a FlowFile to a samba network location. " +
+ "Use this processor instead of a cifs mounts if share access control is important.")
+@SeeAlso({GetSmbFile.class})
+@ReadsAttributes({@ReadsAttribute(attribute="filename", description="The filename to use when writing the FlowFile to the network folder.")})
+public class PutSmbFile extends AbstractProcessor {
+ public static final String SHARE_ACCESS_NONE = "none";
+ public static final String SHARE_ACCESS_READ = "read";
+ public static final String SHARE_ACCESS_READDELETE = "read, delete";
+ public static final String SHARE_ACCESS_READWRITEDELETE = "read, write, delete";
+
+ public static final String REPLACE_RESOLUTION = "replace";
+ public static final String IGNORE_RESOLUTION = "ignore";
+ public static final String FAIL_RESOLUTION = "fail";
+
+ public static final PropertyDescriptor HOSTNAME = new PropertyDescriptor.Builder()
+ .name("Hostname")
+ .description("The network host to which files should be written.")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor SHARE = new PropertyDescriptor.Builder()
+ .name("Share")
+ .description("The network share to which files should be written.")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor DIRECTORY = new PropertyDescriptor.Builder()
+ .name("Directory")
+ .description("The network folder to which files should be written. You may use expression language.")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor DOMAIN = new PropertyDescriptor.Builder()
+ .name("Domain")
+ .description("The domain use for authentication")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
+ .name("Username")
+ .description("The username use for authentication")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
+ .name("Password")
+ .description("The password use for authentication")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor CREATE_DIRS = new PropertyDescriptor.Builder()
+ .name("Create Missing Directories")
+ .description("If true, then missing destination directories will be created. If false, flowfiles are penalized and sent to failure.")
+ .required(true)
+ .allowableValues("true", "false")
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor SHARE_ACCESS = new PropertyDescriptor.Builder()
+ .name("Share Access Strategy")
+ .description("Indicates which shared access are granted on the file during the write. " +
+ "None is the most restrictive, but the safest setting to prevent corruption.")
+ .required(true)
+ .defaultValue(SHARE_ACCESS_NONE)
+ .allowableValues(SHARE_ACCESS_NONE, SHARE_ACCESS_READ, SHARE_ACCESS_READDELETE, SHARE_ACCESS_READWRITEDELETE)
+ .build();
+ public static final PropertyDescriptor CONFLICT_RESOLUTION = new PropertyDescriptor.Builder()
+ .name("Conflict Resolution Strategy")
+ .description("Indicates what should happen when a file with the same name already exists in the output directory")
+ .required(true)
+ .defaultValue(REPLACE_RESOLUTION)
+ .allowableValues(REPLACE_RESOLUTION, IGNORE_RESOLUTION, FAIL_RESOLUTION)
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .description("The maximum number of files to put in each iteration")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("100")
+ .build();
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Files that have been successfully written to the output network path are transferred to this relationship")
+ .build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Files that could not be written to the output network path for some reason are transferred to this relationship")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ private SMBClient smbClient = null;
Review comment:
The connect method is synchronized in [the source code](https://github.com/hierynomus/smbj/blob/a8c74679c977946ff99a68cd19e69c6b20cf93d6/src/main/java/com/hierynomus/smbj/SMBClient.java#L96). So I suppose there is no need for triggerSerially.
That being said I couldn't find anything about thread safety in the docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] joewitt commented on issue #4169: NIFI-7338 Adding
GetSmbFile and PutSmbFile processors #2
Posted by GitBox <gi...@apache.org>.
joewitt commented on issue #4169: NIFI-7338 Adding GetSmbFile and PutSmbFile processors #2
URL: https://github.com/apache/nifi/pull/4169#issuecomment-611089045
I am reviewing this for now. But the level of effort to do so is problematic.
First I dont have any SMB file shares enabled so I'm reading how to set that up. The processor docs don't cover any details on the configuration scenarios. workgroup, user and password are all not required. What combination is that good for? What if I only set a workgroup? What if workgroup and user but not password? There should be descriptions of these cases.
Have more questions there I think.
does this processor set the provenance values correctly?
Nar size is 4MB so that is good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services