You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/12/22 16:29:58 UTC
[jira] [Commented] (NIFI-1856) ExecuteStreamCommand Needs to
Consume Standard Error
[ https://issues.apache.org/jira/browse/NIFI-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770439#comment-15770439 ]
ASF GitHub Bot commented on NIFI-1856:
--------------------------------------
GitHub user rkarthik29 opened a pull request:
https://github.com/apache/nifi/pull/1355
NIFI-1856 ExecuteStreamCommand Needs to Consume Standard Error
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rkarthik29/nifi-1 master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1355.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1355
----
commit 862292f7ccba3448831a17b363fe88ab5872aca0
Author: Kathik Narayanan <kn...@hortonworks.com>
Date: 2016-11-21T01:18:45Z
NiFI-3604 onvert to Avro fails when a MySQL query returns two columns with the same name even with different aliases
commit d8d598b3472d709349eb3d25f8d10cbad8be556b
Author: Kathik Narayanan <kn...@hortonworks.com>
Date: 2016-11-30T20:29:32Z
Merge branch 'master' of https://github.com/apache/nifi
commit 417f944173c9d155f8f38b9f803a58eaaca5518d
Author: Kathik Narayanan <kn...@hortonworks.com>
Date: 2016-12-12T14:47:09Z
Merge branch 'master' of https://github.com/apache/nifi
commit b7bf487ba743eb9a6664d06b399e2712eeba05fd
Author: Kathik Narayanan <kn...@hortonworks.com>
Date: 2016-12-19T19:11:31Z
Merge branch 'master' of https://github.com/apache/nifi
commit 384325fe6a1965d435ced0ec1afb75a6ef9a0bd6
Author: Kathik Narayanan <kn...@hortonworks.com>
Date: 2016-12-22T05:30:26Z
Nifi-1856 ExecuteStreamCommand Needs to Consume Standard Error
commit 131869a8e22af78c366691e8944ef57c49bb53c6
Author: Kathik Narayanan <kn...@hortonworks.com>
Date: 2016-12-22T16:04:44Z
Merge branch 'master' of https://github.com/apache/nifi
commit 27fb13e4d274ba83ce0873457f103bb0f4d54316
Author: Kathik Narayanan <kn...@hortonworks.com>
Date: 2016-12-22T16:05:26Z
Merge branch 'nifi-1856'
----
> ExecuteStreamCommand Needs to Consume Standard Error
> ----------------------------------------------------
>
> Key: NIFI-1856
> URL: https://issues.apache.org/jira/browse/NIFI-1856
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Alan Jackoway
> Assignee: Karthik Narayanan
>
> I was using ExecuteStreamProcess to run certain hdfs commands that are tricky to write in nifi but easy in bash (e.g. {{hadoop fs -rm -r /data/*/2014/05/05}})
> However, my larger commands kept hanging even though when I run them from the command line they finish quickly.
> Based on http://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html I believe that ExecuteStreamCommand and possibly other processors need to consume the standard error stream to prevent the processes from blocking when standard error gets filled.
> To reproduce. Create this as ~/write.py
> {code:python}
> import sys
> count = int(sys.argv[1])
> for x in range(count):
> sys.stderr.write("ERROR %d\n" % x)
> sys.stdout.write("OUTPUT %d\n" % x)
> {code}
> Create a flow that goes
> # GenerateFlowFile - 5 minutes schedule 0 bytes size
> # ExecuteStreamCommand - Command arguments /Users/alanj/write.py;100 Command Path python
> # PutFile - /tmp/write/
> routing output stream of ExecuteStreamCommand to PutFile
> When you turn everything on, you get 100 lines (not 200) of just the standard output in /tmp/write.
> Next, change the command arguments to /Users/alanj/write.py;100000 and turn everything on again. The command will hang.
> I believe that whenever you execute a process the way ExecuteStreamCommand is doing, you need to consume the standard error stream to keep it from blocking. This may also affect things like ExecuteProcess and ExecuteScript as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)