You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by pvillard31 <gi...@git.apache.org> on 2016/11/03 20:56:21 UTC
[GitHub] nifi pull request #1181: NIFI-1322 - PutHDFS - allow file append resolution
GitHub user pvillard31 opened a pull request:
https://github.com/apache/nifi/pull/1181
NIFI-1322 - PutHDFS - allow file append resolution
Two remarks:
- LocalFileSystem object used for unit tests does not implement the append method.
- Based on HDFS-4600 (https://issues.apache.org/jira/browse/HDFS-4600), appending to a file is only possible if the replication factor is lesser or equal than the number of data nodes in the cluster. For live testing with a standalone instance, consider changing the replication factor to 1 for the file you are appending to.
Also, I considered that it would be up to the user to handle the demarcator between two appended messages if such a demarcator is expected.
Finally, if two processors are appending to the same file, errors may be raised due to concurrent access to the file. However, if redirecting the failure relationship back to the processor, the processors should, in the end, be able to append the messages. Obviously, in such a case, message ordering shall not be a requirement. Local testing suggests that users should use a MergeContent processor as upstream processor in order to ensure acceptable bandwidth.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pvillard31/nifi NIFI-1322
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1181.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1181
----
commit a42ed1d547762b51dbe27275aa54e0736e7046d4
Author: Pierre Villard <pi...@gmail.com>
Date: 2016-11-02T19:38:52Z
PutHDFS - allow file append resolution
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1181: NIFI-1322 - PutHDFS - allow file append resolution
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1181#discussion_r86523527
--- Diff: nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java ---
@@ -74,9 +75,15 @@
@SeeAlso(GetHDFS.class)
public class PutHDFS extends AbstractHadoopProcessor {
- public static final String REPLACE_RESOLUTION = "replace";
- public static final String IGNORE_RESOLUTION = "ignore";
- public static final String FAIL_RESOLUTION = "fail";
--- End diff --
Pierre; I understand why/what you did here, but given the fact that both the class and the variable names are ```public```, this constitutes a breaking change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1181: NIFI-1322 - PutHDFS - allow file append resolution
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1181#discussion_r86524045
--- Diff: nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java ---
@@ -315,21 +328,24 @@ public void process(InputStream in) throws IOException {
final long millis = stopWatch.getDuration(TimeUnit.MILLISECONDS);
tempDotCopyFile = tempCopyFile;
- boolean renamed = false;
- for (int i = 0; i < 10; i++) { // try to rename multiple times.
- if (hdfs.rename(tempCopyFile, copyFile)) {
- renamed = true;
- break;// rename was successful
+ if(!conflictResponse.equals(APPEND_RESOLUTION.getValue())
+ || (conflictResponse.equals(APPEND_RESOLUTION.getValue()) && !destinationExists)) {
+ boolean renamed = false;
+ for (int i = 0; i < 10; i++) { // try to rename multiple times.
+ if (hdfs.rename(tempCopyFile, copyFile)) {
+ renamed = true;
+ break;// rename was successful
+ }
+ Thread.sleep(200L);// try waiting to let whatever might cause rename failure to resolve
--- End diff --
Do you know what's this for and if we still need it? I mean not only the sleep, but he whole loop.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1181: NIFI-1322 - PutHDFS - allow file append resolution
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1181#discussion_r86528178
--- Diff: nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java ---
@@ -315,21 +328,24 @@ public void process(InputStream in) throws IOException {
final long millis = stopWatch.getDuration(TimeUnit.MILLISECONDS);
tempDotCopyFile = tempCopyFile;
- boolean renamed = false;
- for (int i = 0; i < 10; i++) { // try to rename multiple times.
- if (hdfs.rename(tempCopyFile, copyFile)) {
- renamed = true;
- break;// rename was successful
+ if(!conflictResponse.equals(APPEND_RESOLUTION.getValue())
+ || (conflictResponse.equals(APPEND_RESOLUTION.getValue()) && !destinationExists)) {
+ boolean renamed = false;
+ for (int i = 0; i < 10; i++) { // try to rename multiple times.
+ if (hdfs.rename(tempCopyFile, copyFile)) {
+ renamed = true;
+ break;// rename was successful
+ }
+ Thread.sleep(200L);// try waiting to let whatever might cause rename failure to resolve
--- End diff --
To be honest, no I don't. I assume that there is some kind of asynchronous call we are doing before when talking to remote clusters and that we might need to make this try multiple times to actually succeed. Not sure if it is still required with the last versions of HDFS client but didn't want to change it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1181: NIFI-1322 - PutHDFS - allow file append resolution
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/1181
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1181: NIFI-1322 - PutHDFS - allow file append resolution
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1181#discussion_r86526545
--- Diff: nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java ---
@@ -74,9 +75,15 @@
@SeeAlso(GetHDFS.class)
public class PutHDFS extends AbstractHadoopProcessor {
- public static final String REPLACE_RESOLUTION = "replace";
- public static final String IGNORE_RESOLUTION = "ignore";
- public static final String FAIL_RESOLUTION = "fail";
--- End diff --
Oh yes, you're right! Will update this part asap!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---