You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@edgent.apache.org by wmarshall484 <gi...@git.apache.org> on 2016/03/22 00:12:15 UTC
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
GitHub user wmarshall484 opened a pull request:
https://github.com/apache/incubator-quarks-website/pull/13
Added Quarks cookbook, hello_quarks, and source_function recipes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wmarshall484/incubator-quarks-website master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-quarks-website/pull/13.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13
----
commit 2230ca40942e5b8ff02814da1679dc69bb72bcf0
Author: wcmarsha <wc...@us.ibm.com>
Date: 2016-03-21T23:06:02Z
Added Quarks cookbook, hello_quarks, and source_function recipes.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by ddebrunner <gi...@git.apache.org>.
Github user ddebrunner commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r56917803
--- Diff: site/docs/recipe_source_function.md ---
@@ -0,0 +1,86 @@
+---
+title: Recipe 2. Writing a Source Function
+---
+In the previous [Hello Quarks!](recipe_hello_quarks) example, we create a data source which only generates a single Java String and prints it to output. Yet Quarks sources support the ability generate any data type as a source, not just primitive Java types such as Strings. Moreover, because the user supplies the code which generates the data, the user has complete flexibility for *how* the data is generated. This recipe demonstrates how a user could write such a custom data source.
--- End diff --
Nit: A String is not a Java primitive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by wmarshall484 <gi...@git.apache.org>.
Github user wmarshall484 commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r57074967
--- Diff: site/docs/recipe_source_function.md ---
@@ -0,0 +1,86 @@
+---
+title: Recipe 2. Writing a Source Function
+---
+In the previous [Hello Quarks!](recipe_hello_quarks) example, we create a data source which only generates a single Java String and prints it to output. Yet Quarks sources support the ability generate any data type as a source, not just primitive Java types such as Strings. Moreover, because the user supplies the code which generates the data, the user has complete flexibility for *how* the data is generated. This recipe demonstrates how a user could write such a custom data source.
+
+## Custom Source: Reading the Lines of a Web Page
+{{site.data.alerts.note}} Quarks' API provides convenience methods for performing HTTP requests. For the sake of example we are writing a HTTP data source manually, but in principle there are easier methods. {{site.data.alerts.end}}
+
+One example of a custom data source could be retrieving the contents of a web page and printing each line to output. For example, the user could be querying the Yahoo Finance website for the most recent stock price data of Bank of America, Cabot Oil & Gas, and Freeport-McMoRan Inc:
+
+``` java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+ }
+```
+
+Given the correctly formatted URL to request the data, we can use the *Topology.source* method to generate each line of the page as a data item on the stream. *Topology.source* takes a Java Supplier that returns an Iterable. The supplier is invoked once, and the items returned from the Iterable are used as the stream's data items. For example, the following *queryWebsite* method returns a supplier which queries a URL and returns an Iterable of its contents:
+
+``` java
+ private static Supplier<Iterable<String> > queryWebsite(URL url) throws Exception{
+ return () -> {
+ List<String> lines = new LinkedList<>();
+ try {
+ InputStream is = url.openStream();
+ BufferedReader br = new BufferedReader(
+ new InputStreamReader(is));
+
+ for(String s = br.readLine(); s != null; s = br.readLine())
+ lines.add(s);
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return lines;
+ };
+ }
+```
+
+ When invoking *Topology.source*, we can use *queryWebsite* to return the required supplier, passing in the URL.
+
+ ``` java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+
+ TStream<String> linesOfWebsite = top.source(queryWebsite(url));
+}
--- End diff --
Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by queeniema <gi...@git.apache.org>.
Github user queeniema commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r56919503
--- Diff: site/docs/recipe_hello_quarks.md ---
@@ -0,0 +1,62 @@
+---
+title: Recipe 1. Hello Quarks!
+---
+
+Quarks' pure Java implementation is a powerful feature which allows it to be run on the majority of JVM-compatible systems. It also has the added benefit of enabling the developer to develop applications entirely within the Eclipse and Intellij ecosystems. For the purposes of this (and future) recipes, it will be assumed that the developer is using Eclipse. To begin the Hello World recipe, create a new project and import the necessary libraries as outlined in the [getting started guide](quarks-getting-started). Next, write the following template application:
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ }
+```
+
+The *DirectProvider* is an object which allows the user to submit and run the final application. It also creates the *Topology* object, which gives the developer the ability to define a stream of strings.
+
+## Using Topology.strings
+The primary abstraction in Quarks is the *TStream*. A *TStream* represents the flow of data in a Quarks application; for example, the periodic floating point readings from a temperature sensor. The data items which are sent through a *TStream* are Java objects -- in the "Hello Quarks!" example, we are sending a single java.lang.String. There are a number of ways to create a *TStream*, and *Topology.strings* is the simplest. The user specifies a number of strings which will be used as the stream's data items.
+
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ TStream<String> helloStream = top.strings("Hello Quarks!");
+ }
+```
+
+The *helloStream* stream is created, and the "Hello Quarks!" string will be sent as its single data item.
+## Printing to Output
+*TStream.print* can be used to print the data items of a stream to standard output by invoking the *toString* method of each data item. In this case the data items are already strings, but in principle *TStream.print* can be called on any stream, regardless of the datatype carried by the stream.
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ TStream<String> helloStream = top.strings("Hello Quarks!");
+ helloStream.print();
+ }
+```
+
+## Submitting the Application
+The only remaining step is to submit the application, which is performed my the *DirectProvider*. Submitting a Quarks application initializes the threads which execute the *Topology*, and begins processing its data sources.
--- End diff --
Typo: Should "my" be "by"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by wmarshall484 <gi...@git.apache.org>.
Github user wmarshall484 commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r57075008
--- Diff: site/docs/recipe_hello_quarks.md ---
@@ -0,0 +1,62 @@
+---
+title: Recipe 1. Hello Quarks!
+---
+
+Quarks' pure Java implementation is a powerful feature which allows it to be run on the majority of JVM-compatible systems. It also has the added benefit of enabling the developer to develop applications entirely within the Eclipse and Intellij ecosystems. For the purposes of this (and future) recipes, it will be assumed that the developer is using Eclipse. To begin the Hello World recipe, create a new project and import the necessary libraries as outlined in the [getting started guide](quarks-getting-started). Next, write the following template application:
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ }
+```
+
+The *DirectProvider* is an object which allows the user to submit and run the final application. It also creates the *Topology* object, which gives the developer the ability to define a stream of strings.
+
+## Using Topology.strings
+The primary abstraction in Quarks is the *TStream*. A *TStream* represents the flow of data in a Quarks application; for example, the periodic floating point readings from a temperature sensor. The data items which are sent through a *TStream* are Java objects -- in the "Hello Quarks!" example, we are sending a single java.lang.String. There are a number of ways to create a *TStream*, and *Topology.strings* is the simplest. The user specifies a number of strings which will be used as the stream's data items.
+
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ TStream<String> helloStream = top.strings("Hello Quarks!");
+ }
+```
+
+The *helloStream* stream is created, and the "Hello Quarks!" string will be sent as its single data item.
+## Printing to Output
+*TStream.print* can be used to print the data items of a stream to standard output by invoking the *toString* method of each data item. In this case the data items are already strings, but in principle *TStream.print* can be called on any stream, regardless of the datatype carried by the stream.
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ TStream<String> helloStream = top.strings("Hello Quarks!");
+ helloStream.print();
+ }
+```
+
+## Submitting the Application
+The only remaining step is to submit the application, which is performed my the *DirectProvider*. Submitting a Quarks application initializes the threads which execute the *Topology*, and begins processing its data sources.
--- End diff --
Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by wmarshall484 <gi...@git.apache.org>.
Github user wmarshall484 commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r57074944
--- Diff: site/docs/recipe_source_function.md ---
@@ -0,0 +1,86 @@
+---
+title: Recipe 2. Writing a Source Function
+---
+In the previous [Hello Quarks!](recipe_hello_quarks) example, we create a data source which only generates a single Java String and prints it to output. Yet Quarks sources support the ability generate any data type as a source, not just primitive Java types such as Strings. Moreover, because the user supplies the code which generates the data, the user has complete flexibility for *how* the data is generated. This recipe demonstrates how a user could write such a custom data source.
--- End diff --
Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by wmarshall484 <gi...@git.apache.org>.
Github user wmarshall484 commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r57074934
--- Diff: site/docs/recipe_hello_quarks.md ---
@@ -0,0 +1,62 @@
+---
+title: Recipe 1. Hello Quarks!
+---
+
+Quarks' pure Java implementation is a powerful feature which allows it to be run on the majority of JVM-compatible systems. It also has the added benefit of enabling the developer to develop applications entirely within the Eclipse and Intellij ecosystems. For the purposes of this (and future) recipes, it will be assumed that the developer is using Eclipse. To begin the Hello World recipe, create a new project and import the necessary libraries as outlined in the [getting started guide](quarks-getting-started). Next, write the following template application:
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ }
+```
+
+The *DirectProvider* is an object which allows the user to submit and run the final application. It also creates the *Topology* object, which gives the developer the ability to define a stream of strings.
--- End diff --
Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by ddebrunner <gi...@git.apache.org>.
Github user ddebrunner commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r56917260
--- Diff: site/docs/recipe_hello_quarks.md ---
@@ -0,0 +1,62 @@
+---
+title: Recipe 1. Hello Quarks!
+---
+
+Quarks' pure Java implementation is a powerful feature which allows it to be run on the majority of JVM-compatible systems. It also has the added benefit of enabling the developer to develop applications entirely within the Eclipse and Intellij ecosystems. For the purposes of this (and future) recipes, it will be assumed that the developer is using Eclipse. To begin the Hello World recipe, create a new project and import the necessary libraries as outlined in the [getting started guide](quarks-getting-started). Next, write the following template application:
--- End diff --
I put a comment in QUARKS-16
Looks good though maybe we should not have all recipes directly in the `docs` directory, but at least a `recipes` or `cookbook` folder and maybe a sub-directory under that for each recipe?
Image if we had 100 recipes ...
https://issues.apache.org/jira/browse/QUARKS-16?focusedCommentId=15205345&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15205345
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-quarks-website/pull/13
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by wmarshall484 <gi...@git.apache.org>.
Github user wmarshall484 commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r57074986
--- Diff: site/docs/recipe_source_function.md ---
@@ -0,0 +1,86 @@
+---
+title: Recipe 2. Writing a Source Function
+---
+In the previous [Hello Quarks!](recipe_hello_quarks) example, we create a data source which only generates a single Java String and prints it to output. Yet Quarks sources support the ability generate any data type as a source, not just primitive Java types such as Strings. Moreover, because the user supplies the code which generates the data, the user has complete flexibility for *how* the data is generated. This recipe demonstrates how a user could write such a custom data source.
+
+## Custom Source: Reading the Lines of a Web Page
+{{site.data.alerts.note}} Quarks' API provides convenience methods for performing HTTP requests. For the sake of example we are writing a HTTP data source manually, but in principle there are easier methods. {{site.data.alerts.end}}
+
+One example of a custom data source could be retrieving the contents of a web page and printing each line to output. For example, the user could be querying the Yahoo Finance website for the most recent stock price data of Bank of America, Cabot Oil & Gas, and Freeport-McMoRan Inc:
+
+``` java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+ }
+```
+
+Given the correctly formatted URL to request the data, we can use the *Topology.source* method to generate each line of the page as a data item on the stream. *Topology.source* takes a Java Supplier that returns an Iterable. The supplier is invoked once, and the items returned from the Iterable are used as the stream's data items. For example, the following *queryWebsite* method returns a supplier which queries a URL and returns an Iterable of its contents:
+
+``` java
+ private static Supplier<Iterable<String> > queryWebsite(URL url) throws Exception{
+ return () -> {
+ List<String> lines = new LinkedList<>();
+ try {
+ InputStream is = url.openStream();
+ BufferedReader br = new BufferedReader(
+ new InputStreamReader(is));
+
+ for(String s = br.readLine(); s != null; s = br.readLine())
+ lines.add(s);
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return lines;
+ };
+ }
+```
+
+ When invoking *Topology.source*, we can use *queryWebsite* to return the required supplier, passing in the URL.
+
+ ``` java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+
+ TStream<String> linesOfWebsite = top.source(queryWebsite(url));
+}
+ ```
+
+ Source methods such as *Topology.source* and *Topology.strings* return a *TStream*. If we print the *linesOfWebsite* stream to standard output and run the application, we can see that it correctly generates the data and feeds it into the Quarks runtime:
+
+Output:
+
+```java
+"BAC","Bank of America Corporation Com",13.150,13.140,"12:00pm - <b>13.145</b>"
+"COG","Cabot Oil & Gas Corporation Com",21.6800,21.6700,"12:00pm - <b>21.6775</b>"
+"FCX","Freeport-McMoRan, Inc. Common S",8.8200,8.8100,"12:00pm - <b>8.8035</b>"
+```
+
+## Polling source: reading data periodically
+A much more common scenario for a developer is the periodic generation of data from a source operator -- a data source may need to be polled every 5 seconds, 3 hours, or any time frame. To this end, *Topology* exposes the *poll* method which can be used to call a function at the frequency of the user's choosing. For example, a user might want to query Yahoo Finance every two seconds to retrieve the most up to date ticker price for a stock:
+
+```java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+
+ TStream<Iterable<String>> source = top.poll(queryWebsite(url), 2, TimeUnit.SECONDS);
+ source.print();
+
+ dp.submit(top);
+ }
+```
+
+**Output:**
+<br>
+<img src="images/pollingSource.gif">
+
+It's important to note that calls to *DirectProvider.submit* are non-blocking; the main thread will exit, and the threads executing the topology will continue to run. (Also, to see changing stock prices, the above example needs to be run during open trading hours. Otherwise, it will simple return the same results every time the website is polled).
--- End diff --
Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by queeniema <gi...@git.apache.org>.
Github user queeniema commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r56916886
--- Diff: site/docs/recipe_hello_quarks.md ---
@@ -0,0 +1,62 @@
+---
+title: Recipe 1. Hello Quarks!
+---
+
+Quarks' pure Java implementation is a powerful feature which allows it to be run on the majority of JVM-compatible systems. It also has the added benefit of enabling the developer to develop applications entirely within the Eclipse and Intellij ecosystems. For the purposes of this (and future) recipes, it will be assumed that the developer is using Eclipse. To begin the Hello World recipe, create a new project and import the necessary libraries as outlined in the [getting started guide](quarks-getting-started). Next, write the following template application:
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ }
+```
+
+The *DirectProvider* is an object which allows the user to submit and run the final application. It also creates the *Topology* object, which gives the developer the ability to define a stream of strings.
--- End diff --
Would it be better to have API references as inline code blocks? For example, `DirectProvider` instead of *DirectProvider*?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by queeniema <gi...@git.apache.org>.
Github user queeniema commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r56918421
--- Diff: site/docs/recipe_source_function.md ---
@@ -0,0 +1,86 @@
+---
+title: Recipe 2. Writing a Source Function
+---
+In the previous [Hello Quarks!](recipe_hello_quarks) example, we create a data source which only generates a single Java String and prints it to output. Yet Quarks sources support the ability generate any data type as a source, not just primitive Java types such as Strings. Moreover, because the user supplies the code which generates the data, the user has complete flexibility for *how* the data is generated. This recipe demonstrates how a user could write such a custom data source.
+
+## Custom Source: Reading the Lines of a Web Page
+{{site.data.alerts.note}} Quarks' API provides convenience methods for performing HTTP requests. For the sake of example we are writing a HTTP data source manually, but in principle there are easier methods. {{site.data.alerts.end}}
+
+One example of a custom data source could be retrieving the contents of a web page and printing each line to output. For example, the user could be querying the Yahoo Finance website for the most recent stock price data of Bank of America, Cabot Oil & Gas, and Freeport-McMoRan Inc:
+
+``` java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+ }
+```
+
+Given the correctly formatted URL to request the data, we can use the *Topology.source* method to generate each line of the page as a data item on the stream. *Topology.source* takes a Java Supplier that returns an Iterable. The supplier is invoked once, and the items returned from the Iterable are used as the stream's data items. For example, the following *queryWebsite* method returns a supplier which queries a URL and returns an Iterable of its contents:
+
+``` java
+ private static Supplier<Iterable<String> > queryWebsite(URL url) throws Exception{
+ return () -> {
+ List<String> lines = new LinkedList<>();
+ try {
+ InputStream is = url.openStream();
+ BufferedReader br = new BufferedReader(
+ new InputStreamReader(is));
+
+ for(String s = br.readLine(); s != null; s = br.readLine())
+ lines.add(s);
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return lines;
+ };
+ }
+```
+
+ When invoking *Topology.source*, we can use *queryWebsite* to return the required supplier, passing in the URL.
+
+ ``` java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+
+ TStream<String> linesOfWebsite = top.source(queryWebsite(url));
+}
--- End diff --
Indentation is a bit off here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by wmarshall484 <gi...@git.apache.org>.
Github user wmarshall484 commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r57074824
--- Diff: site/_data/mydoc/mydoc_sidebar.yml ---
@@ -74,6 +74,29 @@ entries:
version: all
output: web, pdf
+ - title: Quarks Cookbook
+ audience: writers, designers
+ platform: all
+ product: all
+ version: all
+ output: web, pdf
+ items:
+ - title: Recipe 1. Hello Quarks
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by queeniema <gi...@git.apache.org>.
Github user queeniema commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r56919226
--- Diff: site/docs/recipe_source_function.md ---
@@ -0,0 +1,86 @@
+---
+title: Recipe 2. Writing a Source Function
+---
+In the previous [Hello Quarks!](recipe_hello_quarks) example, we create a data source which only generates a single Java String and prints it to output. Yet Quarks sources support the ability generate any data type as a source, not just primitive Java types such as Strings. Moreover, because the user supplies the code which generates the data, the user has complete flexibility for *how* the data is generated. This recipe demonstrates how a user could write such a custom data source.
+
+## Custom Source: Reading the Lines of a Web Page
+{{site.data.alerts.note}} Quarks' API provides convenience methods for performing HTTP requests. For the sake of example we are writing a HTTP data source manually, but in principle there are easier methods. {{site.data.alerts.end}}
+
+One example of a custom data source could be retrieving the contents of a web page and printing each line to output. For example, the user could be querying the Yahoo Finance website for the most recent stock price data of Bank of America, Cabot Oil & Gas, and Freeport-McMoRan Inc:
+
+``` java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+ }
+```
+
+Given the correctly formatted URL to request the data, we can use the *Topology.source* method to generate each line of the page as a data item on the stream. *Topology.source* takes a Java Supplier that returns an Iterable. The supplier is invoked once, and the items returned from the Iterable are used as the stream's data items. For example, the following *queryWebsite* method returns a supplier which queries a URL and returns an Iterable of its contents:
+
+``` java
+ private static Supplier<Iterable<String> > queryWebsite(URL url) throws Exception{
+ return () -> {
+ List<String> lines = new LinkedList<>();
+ try {
+ InputStream is = url.openStream();
+ BufferedReader br = new BufferedReader(
+ new InputStreamReader(is));
+
+ for(String s = br.readLine(); s != null; s = br.readLine())
+ lines.add(s);
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return lines;
+ };
+ }
+```
+
+ When invoking *Topology.source*, we can use *queryWebsite* to return the required supplier, passing in the URL.
+
+ ``` java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+
+ TStream<String> linesOfWebsite = top.source(queryWebsite(url));
+}
+ ```
+
+ Source methods such as *Topology.source* and *Topology.strings* return a *TStream*. If we print the *linesOfWebsite* stream to standard output and run the application, we can see that it correctly generates the data and feeds it into the Quarks runtime:
+
+Output:
+
+```java
+"BAC","Bank of America Corporation Com",13.150,13.140,"12:00pm - <b>13.145</b>"
+"COG","Cabot Oil & Gas Corporation Com",21.6800,21.6700,"12:00pm - <b>21.6775</b>"
+"FCX","Freeport-McMoRan, Inc. Common S",8.8200,8.8100,"12:00pm - <b>8.8035</b>"
+```
+
+## Polling source: reading data periodically
+A much more common scenario for a developer is the periodic generation of data from a source operator -- a data source may need to be polled every 5 seconds, 3 hours, or any time frame. To this end, *Topology* exposes the *poll* method which can be used to call a function at the frequency of the user's choosing. For example, a user might want to query Yahoo Finance every two seconds to retrieve the most up to date ticker price for a stock:
+
+```java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+
+ TStream<Iterable<String>> source = top.poll(queryWebsite(url), 2, TimeUnit.SECONDS);
+ source.print();
+
+ dp.submit(top);
+ }
+```
+
+**Output:**
+<br>
+<img src="images/pollingSource.gif">
+
+It's important to note that calls to *DirectProvider.submit* are non-blocking; the main thread will exit, and the threads executing the topology will continue to run. (Also, to see changing stock prices, the above example needs to be run during open trading hours. Otherwise, it will simple return the same results every time the website is polled).
--- End diff --
Typo: "simple" should be "simply"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by queeniema <gi...@git.apache.org>.
Github user queeniema commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r56916488
--- Diff: site/_data/mydoc/mydoc_sidebar.yml ---
@@ -74,6 +74,29 @@ entries:
version: all
output: web, pdf
+ - title: Quarks Cookbook
+ audience: writers, designers
+ platform: all
+ product: all
+ version: all
+ output: web, pdf
+ items:
+ - title: Recipe 1. Hello Quarks
--- End diff --
Should the title be `Hello Quarks!` (i.e., with an exclamation point)? I see that other references to the recipe (e.g., in Recipe 2) include the exclamation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by wmarshall484 <gi...@git.apache.org>.
Github user wmarshall484 commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r57074912
--- Diff: site/docs/recipe_hello_quarks.md ---
@@ -0,0 +1,62 @@
+---
+title: Recipe 1. Hello Quarks!
+---
+
+Quarks' pure Java implementation is a powerful feature which allows it to be run on the majority of JVM-compatible systems. It also has the added benefit of enabling the developer to develop applications entirely within the Eclipse and Intellij ecosystems. For the purposes of this (and future) recipes, it will be assumed that the developer is using Eclipse. To begin the Hello World recipe, create a new project and import the necessary libraries as outlined in the [getting started guide](quarks-getting-started). Next, write the following template application:
--- End diff --
Recipes added to a top-level "recipes" folder. The folder needs to be top-level, otherwise it breaks the paths to the css layouts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: Added Quarks cookbook, hell...
Posted by queeniema <gi...@git.apache.org>.
Github user queeniema commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/13#discussion_r56916617
--- Diff: site/docs/recipe_hello_quarks.md ---
@@ -0,0 +1,62 @@
+---
+title: Recipe 1. Hello Quarks!
+---
+
+Quarks' pure Java implementation is a powerful feature which allows it to be run on the majority of JVM-compatible systems. It also has the added benefit of enabling the developer to develop applications entirely within the Eclipse and Intellij ecosystems. For the purposes of this (and future) recipes, it will be assumed that the developer is using Eclipse. To begin the Hello World recipe, create a new project and import the necessary libraries as outlined in the [getting started guide](quarks-getting-started). Next, write the following template application:
--- End diff --
To be consistent with the wording in the side bar, I would recommend changing "getting started guide" to "Getting Started Guide".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---