You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by lyonbrw <gi...@git.apache.org> on 2015/07/16 19:51:13 UTC
[GitHub] storm pull request: Add FluxParser method parseInputStream() to el...
GitHub user lyonbrw opened a pull request:
https://github.com/apache/storm/pull/637
Add FluxParser method parseInputStream() to eliminate disk usage.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lyonbrw/storm master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/637.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #637
----
commit 9977d5b451fa0e1226876f72035f0dfc68941a5b
Author: Brendan W. Lyon <bl...@datainterfuse.com>
Date: 2015-07-16T17:42:28Z
Merge pull request #1 from apache/master
Update from original
commit 8f88f0864bfd3d2fd25626e592a8b5918d78953c
Author: Brendan W. Lyon <bl...@datainterfuse.com>
Date: 2015-07-16T17:48:00Z
Added parseInputStream() method
Included reusable parseInputStream for use by parseFile and parseResource. Also allows a path for programatically creating topologies without the need to write to / read from disk.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-942] Add FluxParser method parseInputSt...
Posted by lyonbrw <gi...@git.apache.org>.
Github user lyonbrw commented on the pull request:
https://github.com/apache/storm/pull/637#issuecomment-122130478
Will do. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Add FluxParser method parseInputStream() to el...
Posted by lyonbrw <gi...@git.apache.org>.
GitHub user lyonbrw reopened a pull request:
https://github.com/apache/storm/pull/637
Add FluxParser method parseInputStream() to eliminate disk usage.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lyonbrw/storm master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/637.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #637
----
commit 9977d5b451fa0e1226876f72035f0dfc68941a5b
Author: Brendan W. Lyon <bl...@datainterfuse.com>
Date: 2015-07-16T17:42:28Z
Merge pull request #1 from apache/master
Update from original
commit 8f88f0864bfd3d2fd25626e592a8b5918d78953c
Author: Brendan W. Lyon <bl...@datainterfuse.com>
Date: 2015-07-16T17:48:00Z
Added parseInputStream() method
Included reusable parseInputStream for use by parseFile and parseResource. Also allows a path for programatically creating topologies without the need to write to / read from disk.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-942] Add FluxParser method parseInputSt...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/637#issuecomment-122090455
@lyonbrw
Indentation is still broken. Please compare parseInputStream()'s indent level to other methods.
Btw, could you get rid of merge commit from PR if you don't mind?
It doesn't affect the source code of Storm, but leaves unrelated commit log.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-942] Add FluxParser method parseInputSt...
Posted by lyonbrw <gi...@git.apache.org>.
Github user lyonbrw commented on the pull request:
https://github.com/apache/storm/pull/637#issuecomment-122088647
Just saw that redundant close() function before you commented. My bad. ;) Thanks. Corrected both, build in progress.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-942] Add FluxParser method parseInputSt...
Posted by lyonbrw <gi...@git.apache.org>.
Github user lyonbrw closed the pull request at:
https://github.com/apache/storm/pull/637
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-942] Add FluxParser method parseInputSt...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/637#discussion_r34834409
--- Diff: external/flux/flux-core/src/main/java/org/apache/storm/flux/parser/FluxParser.java ---
@@ -44,40 +44,47 @@ private FluxParser(){}
// TODO refactor input stream processing (see parseResource() method).
public static TopologyDef parseFile(String inputFile, boolean dumpYaml, boolean processIncludes,
String propertiesFile, boolean envSub) throws IOException {
- Yaml yaml = yaml();
+
FileInputStream in = new FileInputStream(inputFile);
- // TODO process properties, etc.
- TopologyDef topology = loadYaml(yaml, in, propertiesFile, envSub);
+ TopologyDef topology = parseInputStream(in, dumpYaml, processIncludes, propertiesFile, envSub);
in.close();
- if(dumpYaml){
- dumpYaml(topology, yaml);
- }
- if(processIncludes) {
- return processIncludes(yaml, topology, propertiesFile, envSub);
- } else {
- return topology;
- }
+
+ return topology;
}
public static TopologyDef parseResource(String resource, boolean dumpYaml, boolean processIncludes,
String propertiesFile, boolean envSub) throws IOException {
- Yaml yaml = yaml();
+
InputStream in = FluxParser.class.getResourceAsStream(resource);
- if(in == null){
- LOG.error("Unable to load classpath resource: " + resource);
- System.exit(1);
- }
- TopologyDef topology = loadYaml(yaml, in, propertiesFile, envSub);
+ TopologyDef topology = parseInputStream(in, dumpYaml, processIncludes, propertiesFile, envSub);
in.close();
- if(dumpYaml){
- dumpYaml(topology, yaml);
- }
- if(processIncludes) {
- return processIncludes(yaml, topology, propertiesFile, envSub);
- } else {
- return topology;
- }
+
+ return topology;
}
+
+ public static TopologyDef parseInputStream(InputStream inputStream, boolean dumpYaml, boolean processIncludes,
+ String propertiesFile, boolean envSub) throws IOException {
+
+ Yaml yaml = yaml();
+
+ if (inputStream == null) {
--- End diff --
@lyonbrw Indentation looks broken.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Add FluxParser method parseInputStream() to el...
Posted by lyonbrw <gi...@git.apache.org>.
Github user lyonbrw closed the pull request at:
https://github.com/apache/storm/pull/637
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-942] Add FluxParser method parseInputSt...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/637#issuecomment-122117993
@lyonbrw
I cherry-picked your efforts into master now. Would you close this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-942] Add FluxParser method parseInputSt...
Posted by lyonbrw <gi...@git.apache.org>.
Github user lyonbrw commented on the pull request:
https://github.com/apache/storm/pull/637#issuecomment-122099783
Indentation fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Add FluxParser method parseInputStream() to el...
Posted by lyonbrw <gi...@git.apache.org>.
Github user lyonbrw commented on the pull request:
https://github.com/apache/storm/pull/637#issuecomment-122034395
Included reusable parseInputStream for use by parseFile and parseResource. Also allows a path for programatically creating topologies without the need to write to / read from disk.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-942] Add FluxParser method parseInputSt...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/637#issuecomment-122099913
@lyonbrw
Actually I recommended you to make new branch before do some works.
I can cherry-pick your commits into master, so let me handle it for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-942] Add FluxParser method parseInputSt...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/637#discussion_r34835109
--- Diff: external/flux/flux-core/src/main/java/org/apache/storm/flux/parser/FluxParser.java ---
@@ -44,40 +44,47 @@ private FluxParser(){}
// TODO refactor input stream processing (see parseResource() method).
public static TopologyDef parseFile(String inputFile, boolean dumpYaml, boolean processIncludes,
String propertiesFile, boolean envSub) throws IOException {
- Yaml yaml = yaml();
+
FileInputStream in = new FileInputStream(inputFile);
- // TODO process properties, etc.
- TopologyDef topology = loadYaml(yaml, in, propertiesFile, envSub);
+ TopologyDef topology = parseInputStream(in, dumpYaml, processIncludes, propertiesFile, envSub);
in.close();
- if(dumpYaml){
- dumpYaml(topology, yaml);
- }
- if(processIncludes) {
- return processIncludes(yaml, topology, propertiesFile, envSub);
- } else {
- return topology;
- }
+
+ return topology;
}
public static TopologyDef parseResource(String resource, boolean dumpYaml, boolean processIncludes,
String propertiesFile, boolean envSub) throws IOException {
- Yaml yaml = yaml();
+
InputStream in = FluxParser.class.getResourceAsStream(resource);
- if(in == null){
- LOG.error("Unable to load classpath resource: " + resource);
- System.exit(1);
- }
- TopologyDef topology = loadYaml(yaml, in, propertiesFile, envSub);
+ TopologyDef topology = parseInputStream(in, dumpYaml, processIncludes, propertiesFile, envSub);
in.close();
- if(dumpYaml){
- dumpYaml(topology, yaml);
- }
- if(processIncludes) {
- return processIncludes(yaml, topology, propertiesFile, envSub);
- } else {
- return topology;
- }
+
+ return topology;
}
+
+ public static TopologyDef parseInputStream(InputStream inputStream, boolean dumpYaml, boolean processIncludes,
+ String propertiesFile, boolean envSub) throws IOException {
+
+ Yaml yaml = yaml();
+
+ if (inputStream == null) {
+ LOG.error("Unable to load input stream");
+ System.exit(1);
+ }
+
+ TopologyDef topology = loadYaml(yaml, inputStream, propertiesFile, envSub);
+ inputStream.close();
--- End diff --
@lyonbrw
IMO it isn't a good way to close InputStream which is passed to a parameter. Caller can take care of it, and parseFile / parseResource already do it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-942] Add FluxParser method parseInputSt...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/637#issuecomment-122100673
OK, confirmed that all things fixed. +1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-942] Add FluxParser method parseInputSt...
Posted by lyonbrw <gi...@git.apache.org>.
Github user lyonbrw commented on the pull request:
https://github.com/apache/storm/pull/637#issuecomment-122097348
Apologize for the merge commit. What is the proper way to remove it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---