You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streams.apache.org by sblackmon <sb...@apache.org> on 2016/10/05 22:17:31 UTC
Re: Getting started / tutorial help
Trevor, others,
I’ve opened a few pull requests containing a Flink example module with integration tests demonstrating five pipelines working end-to-end in local mode.
https://github.com/apache/incubator-streams/pull/289
https://github.com/apache/incubator-streams-examples/pull/12
If you have a chance to code review and replicate this, that would be terrific. I believe this demonstrates clearly that Streams is suitable for embedding within real-time parallel processing frameworks.
As we discussed in Berlin there are a lot of cool possible demos based on feeding social streams into Zeppelin workbooks and/or Flink complex event processing libraries.
Steve
On September 30, 2016 at 9:20:09 AM, Steve Blackmon (steve@blackmon.org) wrote:
Trevor,
I don’t see sample.json in the code tree:
https://github.com/apache/incubator-streams-examples/tree/flink/flink/flink-twitter-collection/src/main/jsonschema
Did you happen create that file while trying to run the job or copy it there inadvertently?
The jsonschema2pojo plugin will parse any .json files in that directory and it will fail if they aren’t proper json schemas.
Please keep me posted on your progress.
Cheers,
Steve
On September 30, 2016 at 9:00:53 AM, Trevor Grant (trevor.d.grant@gmail.com) wrote:
Hey all,
just trying to get started on Apache Streams.
I was trying to build the examples (on the Flink branch), when I
-DskipTests I can get a build but when I do mvn clean package I get the
following errors:
skip non existing resourceDirectory
/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/main/xmlschema
[INFO] skip non existing resourceDirectory
/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/main/resources
[INFO] skip non existing resourceDirectory
/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/site/diagrams
[INFO]
[INFO] --- jsonschema2pojo-maven-plugin:0.4.1:generate (default) @
flink-twitter-collection ---
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] streams-examples-flink ............................ SUCCESS [1.752s]
[INFO] flink-twitter-collection .......................... FAILURE [4.383s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 7.455s
[INFO] Finished at: Fri Sep 30 07:58:14 GMT-06:00 2016
[INFO] Final Memory: 30M/430M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.jsonschema2pojo:jsonschema2pojo-maven-plugin:0.4.1:generate (default)
on project flink-twitter-collection: Execution default of goal
org.jsonschema2pojo:jsonschema2pojo-maven-plugin:0.4.1:generate failed:
Error parsing document:
file:/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/main/jsonschema/sample.json:
Unexpected end-of-input within/between OBJECT entries
[ERROR] at [Source:
file:/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/main/jsonschema/sample.json;
line: 69, column: 2279]
Should I be worried?
tg
Trevor Grant
Data Scientist
https://github.com/rawkintrevo
http://stackexchange.com/users/3002022/rawkintrevo
http://trevorgrant.org
*"Fortunate is he, who is able to know the causes of things." -Virgil*
Re: Getting started / tutorial help
Posted by sblackmon <sb...@apache.org>.
All,
Please take a look at this improvement to the examples portion of the website and share any thoughts.
http://streams.staging.apache.org/site/0.4-incubating-SNAPSHOT/streams-examples/index.html
In contrast to with the current page:
http://streams.incubator.apache.org/site/0.4-incubating-SNAPSHOT/streams-examples/
Major differences are:
- More examples: now 4 flink and 6 local
- Each example module now has one integration test for each pipelines, covering all steps from providers to persisters
- Added a new left sidebar, with a contextual collection of instructions that are relevant to running each example
- HOW-TO’s for launching supporting services for testing with docker maven plugin
- A separate page for each pipeline, even within modules that only have one stream
- A HOCON paradigm using config snippets and includes for supplying configuration required by each pipeline
- Snippets for running pipelines using SBT and Docker (neither of which require building the code first)
This section still needs some clean-up and styling, and could still be way better, but maybe this takes us from 2/10 to 5/10 ?
Are the rest of you able to follow page flow? Is the information supplied adequate and adequately organized for you to successfully run the pipeline with mvn verify, and without?
Steve
On October 5, 2016 at 5:17:32 PM, sblackmon (sblackmon@apache.org) wrote:
Trevor, others,
I’ve opened a few pull requests containing a Flink example module with integration tests demonstrating five pipelines working end-to-end in local mode.
https://github.com/apache/incubator-streams/pull/289
https://github.com/apache/incubator-streams-examples/pull/12
If you have a chance to code review and replicate this, that would be terrific. I believe this demonstrates clearly that Streams is suitable for embedding within real-time parallel processing frameworks.
As we discussed in Berlin there are a lot of cool possible demos based on feeding social streams into Zeppelin workbooks and/or Flink complex event processing libraries.
Steve
On September 30, 2016 at 9:20:09 AM, Steve Blackmon (steve@blackmon.org) wrote:
Trevor,
I don’t see sample.json in the code tree:
https://github.com/apache/incubator-streams-examples/tree/flink/flink/flink-twitter-collection/src/main/jsonschema
Did you happen create that file while trying to run the job or copy it there inadvertently?
The jsonschema2pojo plugin will parse any .json files in that directory and it will fail if they aren’t proper json schemas.
Please keep me posted on your progress.
Cheers,
Steve
On September 30, 2016 at 9:00:53 AM, Trevor Grant (trevor.d.grant@gmail.com) wrote:
Hey all,
just trying to get started on Apache Streams.
I was trying to build the examples (on the Flink branch), when I
-DskipTests I can get a build but when I do mvn clean package I get the
following errors:
skip non existing resourceDirectory
/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/main/xmlschema
[INFO] skip non existing resourceDirectory
/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/main/resources
[INFO] skip non existing resourceDirectory
/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/site/diagrams
[INFO]
[INFO] --- jsonschema2pojo-maven-plugin:0.4.1:generate (default) @
flink-twitter-collection ---
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] streams-examples-flink ............................ SUCCESS [1.752s]
[INFO] flink-twitter-collection .......................... FAILURE [4.383s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 7.455s
[INFO] Finished at: Fri Sep 30 07:58:14 GMT-06:00 2016
[INFO] Final Memory: 30M/430M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.jsonschema2pojo:jsonschema2pojo-maven-plugin:0.4.1:generate (default)
on project flink-twitter-collection: Execution default of goal
org.jsonschema2pojo:jsonschema2pojo-maven-plugin:0.4.1:generate failed:
Error parsing document:
file:/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/main/jsonschema/sample.json:
Unexpected end-of-input within/between OBJECT entries
[ERROR] at [Source:
file:/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/main/jsonschema/sample.json;
line: 69, column: 2279]
Should I be worried?
tg
Trevor Grant
Data Scientist
https://github.com/rawkintrevo
http://stackexchange.com/users/3002022/rawkintrevo
http://trevorgrant.org
*"Fortunate is he, who is able to know the causes of things." -Virgil*