You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@edgent.apache.org by wc...@apache.org on 2016/03/22 22:42:44 UTC
[1/2] incubator-quarks-website git commit: Added Quarks cookbook,
hello_quarks, and source_function recipes.
Repository: incubator-quarks-website
Updated Branches:
refs/heads/master c206d02fa -> 2975c8be5
Added Quarks cookbook, hello_quarks, and source_function recipes.
Project: http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/commit/1f313c62
Tree: http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/tree/1f313c62
Diff: http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/diff/1f313c62
Branch: refs/heads/master
Commit: 1f313c627f544c183dc96e4a3808f348cbb1f73c
Parents: c206d02
Author: wcmarsha <wc...@us.ibm.com>
Authored: Mon Mar 21 16:06:02 2016 -0700
Committer: wcmarsha <wc...@us.ibm.com>
Committed: Tue Mar 22 14:38:50 2016 -0700
----------------------------------------------------------------------
site/_data/mydoc/mydoc_sidebar.yml | 23 +++++++++
site/docs/images/pollingSource.gif | Bin 0 -> 61128 bytes
site/docs/recipe_hello_quarks.md | 62 ++++++++++++++++++++++
site/docs/recipe_source_function.md | 86 +++++++++++++++++++++++++++++++
4 files changed, 171 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/blob/1f313c62/site/_data/mydoc/mydoc_sidebar.yml
----------------------------------------------------------------------
diff --git a/site/_data/mydoc/mydoc_sidebar.yml b/site/_data/mydoc/mydoc_sidebar.yml
index 55640c0..974e3a1 100644
--- a/site/_data/mydoc/mydoc_sidebar.yml
+++ b/site/_data/mydoc/mydoc_sidebar.yml
@@ -74,6 +74,29 @@ entries:
version: all
output: web, pdf
+ - title: Quarks Cookbook
+ audience: writers, designers
+ platform: all
+ product: all
+ version: all
+ output: web, pdf
+ items:
+ - title: Recipe 1. Hello Quarks
+ url: /docs/recipe_hello_quarks.html
+ audience: writers, designers
+ platform: all
+ product: all
+ version: all
+ output: web, pdf
+
+ - title: Recipe 2. Writing a Source Function
+ url: /docs/recipe_source_function.html
+ audience: writers, designers
+ platform: all
+ product: all
+ version: all
+ output: web, pdf
+
- title: Sample Programs
audience: writers, designers
platform: all
http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/blob/1f313c62/site/docs/images/pollingSource.gif
----------------------------------------------------------------------
diff --git a/site/docs/images/pollingSource.gif b/site/docs/images/pollingSource.gif
new file mode 100644
index 0000000..bc979c6
Binary files /dev/null and b/site/docs/images/pollingSource.gif differ
http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/blob/1f313c62/site/docs/recipe_hello_quarks.md
----------------------------------------------------------------------
diff --git a/site/docs/recipe_hello_quarks.md b/site/docs/recipe_hello_quarks.md
new file mode 100644
index 0000000..877c777
--- /dev/null
+++ b/site/docs/recipe_hello_quarks.md
@@ -0,0 +1,62 @@
+---
+title: Recipe 1. Hello Quarks!
+---
+
+Quarks' pure Java implementation is a powerful feature which allows it to be run on the majority of JVM-compatible systems. It also has the added benefit of enabling the developer to develop applications entirely within the Eclipse and Intellij ecosystems. For the purposes of this (and future) recipes, it will be assumed that the developer is using Eclipse. To begin the Hello World recipe, create a new project and import the necessary libraries as outlined in the [getting started guide](quarks-getting-started). Next, write the following template application:
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ }
+```
+
+The *DirectProvider* is an object which allows the user to submit and run the final application. It also creates the *Topology* object, which gives the developer the ability to define a stream of strings.
+
+## Using Topology.strings
+The primary abstraction in Quarks is the *TStream*. A *TStream* represents the flow of data in a Quarks application; for example, the periodic floating point readings from a temperature sensor. The data items which are sent through a *TStream* are Java objects -- in the "Hello Quarks!" example, we are sending a single java.lang.String. There are a number of ways to create a *TStream*, and *Topology.strings* is the simplest. The user specifies a number of strings which will be used as the stream's data items.
+
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ TStream<String> helloStream = top.strings("Hello Quarks!");
+ }
+```
+
+The *helloStream* stream is created, and the "Hello Quarks!" string will be sent as its single data item.
+## Printing to Output
+*TStream.print* can be used to print the data items of a stream to standard output by invoking the *toString* method of each data item. In this case the data items are already strings, but in principle *TStream.print* can be called on any stream, regardless of the datatype carried by the stream.
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ TStream<String> helloStream = top.strings("Hello Quarks!");
+ helloStream.print();
+ }
+```
+
+## Submitting the Application
+The only remaining step is to submit the application, which is performed my the *DirectProvider*. Submitting a Quarks application initializes the threads which execute the *Topology*, and begins processing its data sources.
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ TStream<String> helloStream = top.strings("Hello Quarks!");
+ helloStream.print();
+ dp.submit(top);
+ }
+```
+
+After running the application, the output is "Hello Quarks!":
+
+```
+Hello Quarks!
+```
+
+
+
+
http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/blob/1f313c62/site/docs/recipe_source_function.md
----------------------------------------------------------------------
diff --git a/site/docs/recipe_source_function.md b/site/docs/recipe_source_function.md
new file mode 100644
index 0000000..2fed684
--- /dev/null
+++ b/site/docs/recipe_source_function.md
@@ -0,0 +1,86 @@
+---
+title: Recipe 2. Writing a Source Function
+---
+In the previous [Hello Quarks!](recipe_hello_quarks) example, we create a data source which only generates a single Java String and prints it to output. Yet Quarks sources support the ability generate any data type as a source, not just primitive Java types such as Strings. Moreover, because the user supplies the code which generates the data, the user has complete flexibility for *how* the data is generated. This recipe demonstrates how a user could write such a custom data source.
+
+## Custom Source: Reading the Lines of a Web Page
+{{site.data.alerts.note}} Quarks' API provides convenience methods for performing HTTP requests. For the sake of example we are writing a HTTP data source manually, but in principle there are easier methods. {{site.data.alerts.end}}
+
+One example of a custom data source could be retrieving the contents of a web page and printing each line to output. For example, the user could be querying the Yahoo Finance website for the most recent stock price data of Bank of America, Cabot Oil & Gas, and Freeport-McMoRan Inc:
+
+``` java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+ }
+```
+
+Given the correctly formatted URL to request the data, we can use the *Topology.source* method to generate each line of the page as a data item on the stream. *Topology.source* takes a Java Supplier that returns an Iterable. The supplier is invoked once, and the items returned from the Iterable are used as the stream's data items. For example, the following *queryWebsite* method returns a supplier which queries a URL and returns an Iterable of its contents:
+
+``` java
+ private static Supplier<Iterable<String> > queryWebsite(URL url) throws Exception{
+ return () -> {
+ List<String> lines = new LinkedList<>();
+ try {
+ InputStream is = url.openStream();
+ BufferedReader br = new BufferedReader(
+ new InputStreamReader(is));
+
+ for(String s = br.readLine(); s != null; s = br.readLine())
+ lines.add(s);
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return lines;
+ };
+ }
+```
+
+ When invoking *Topology.source*, we can use *queryWebsite* to return the required supplier, passing in the URL.
+
+ ``` java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+
+ TStream<String> linesOfWebsite = top.source(queryWebsite(url));
+}
+ ```
+
+ Source methods such as *Topology.source* and *Topology.strings* return a *TStream*. If we print the *linesOfWebsite* stream to standard output and run the application, we can see that it correctly generates the data and feeds it into the Quarks runtime:
+
+Output:
+
+```java
+"BAC","Bank of America Corporation Com",13.150,13.140,"12:00pm - <b>13.145</b>"
+"COG","Cabot Oil & Gas Corporation Com",21.6800,21.6700,"12:00pm - <b>21.6775</b>"
+"FCX","Freeport-McMoRan, Inc. Common S",8.8200,8.8100,"12:00pm - <b>8.8035</b>"
+```
+
+## Polling source: reading data periodically
+A much more common scenario for a developer is the periodic generation of data from a source operator -- a data source may need to be polled every 5 seconds, 3 hours, or any time frame. To this end, *Topology* exposes the *poll* method which can be used to call a function at the frequency of the user's choosing. For example, a user might want to query Yahoo Finance every two seconds to retrieve the most up to date ticker price for a stock:
+
+```java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+
+ TStream<Iterable<String>> source = top.poll(queryWebsite(url), 2, TimeUnit.SECONDS);
+ source.print();
+
+ dp.submit(top);
+ }
+```
+
+**Output:**
+<br>
+<img src="images/pollingSource.gif">
+
+It's important to note that calls to *DirectProvider.submit* are non-blocking; the main thread will exit, and the threads executing the topology will continue to run. (Also, to see changing stock prices, the above example needs to be run during open trading hours. Otherwise, it will simple return the same results every time the website is polled).
\ No newline at end of file
[2/2] incubator-quarks-website git commit: Addressed comments/fixed
bugs
Posted by wc...@apache.org.
Addressed comments/fixed bugs
Project: http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/commit/2975c8be
Tree: http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/tree/2975c8be
Diff: http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/diff/2975c8be
Branch: refs/heads/master
Commit: 2975c8be58c9556320dd1a4b26ff719af3f7f9fa
Parents: 1f313c6
Author: wcmarsha <wc...@us.ibm.com>
Authored: Tue Mar 22 14:38:42 2016 -0700
Committer: wcmarsha <wc...@us.ibm.com>
Committed: Tue Mar 22 14:38:52 2016 -0700
----------------------------------------------------------------------
site/_data/mydoc/mydoc_sidebar.yml | 6 +-
site/docs/images/pollingSource.gif | Bin 61128 -> 0 bytes
site/docs/recipe_hello_quarks.md | 62 --------------------
site/docs/recipe_source_function.md | 86 ----------------------------
site/recipes/images/pollingSource.gif | Bin 0 -> 61128 bytes
site/recipes/recipe_hello_quarks.md | 62 ++++++++++++++++++++
site/recipes/recipe_source_function.md | 86 ++++++++++++++++++++++++++++
7 files changed, 151 insertions(+), 151 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/blob/2975c8be/site/_data/mydoc/mydoc_sidebar.yml
----------------------------------------------------------------------
diff --git a/site/_data/mydoc/mydoc_sidebar.yml b/site/_data/mydoc/mydoc_sidebar.yml
index 974e3a1..0773442 100644
--- a/site/_data/mydoc/mydoc_sidebar.yml
+++ b/site/_data/mydoc/mydoc_sidebar.yml
@@ -81,8 +81,8 @@ entries:
version: all
output: web, pdf
items:
- - title: Recipe 1. Hello Quarks
- url: /docs/recipe_hello_quarks.html
+ - title: Recipe 1. Hello Quarks!
+ url: /recipes/recipe_hello_quarks.html
audience: writers, designers
platform: all
product: all
@@ -90,7 +90,7 @@ entries:
output: web, pdf
- title: Recipe 2. Writing a Source Function
- url: /docs/recipe_source_function.html
+ url: /recipes/recipe_source_function.html
audience: writers, designers
platform: all
product: all
http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/blob/2975c8be/site/docs/images/pollingSource.gif
----------------------------------------------------------------------
diff --git a/site/docs/images/pollingSource.gif b/site/docs/images/pollingSource.gif
deleted file mode 100644
index bc979c6..0000000
Binary files a/site/docs/images/pollingSource.gif and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/blob/2975c8be/site/docs/recipe_hello_quarks.md
----------------------------------------------------------------------
diff --git a/site/docs/recipe_hello_quarks.md b/site/docs/recipe_hello_quarks.md
deleted file mode 100644
index 877c777..0000000
--- a/site/docs/recipe_hello_quarks.md
+++ /dev/null
@@ -1,62 +0,0 @@
----
-title: Recipe 1. Hello Quarks!
----
-
-Quarks' pure Java implementation is a powerful feature which allows it to be run on the majority of JVM-compatible systems. It also has the added benefit of enabling the developer to develop applications entirely within the Eclipse and Intellij ecosystems. For the purposes of this (and future) recipes, it will be assumed that the developer is using Eclipse. To begin the Hello World recipe, create a new project and import the necessary libraries as outlined in the [getting started guide](quarks-getting-started). Next, write the following template application:
-
-``` java
- public static void main(String[] args) {
- DirectProvider dp = new DirectProvider();
- Topology top = dp.newTopology();
- }
-```
-
-The *DirectProvider* is an object which allows the user to submit and run the final application. It also creates the *Topology* object, which gives the developer the ability to define a stream of strings.
-
-## Using Topology.strings
-The primary abstraction in Quarks is the *TStream*. A *TStream* represents the flow of data in a Quarks application; for example, the periodic floating point readings from a temperature sensor. The data items which are sent through a *TStream* are Java objects -- in the "Hello Quarks!" example, we are sending a single java.lang.String. There are a number of ways to create a *TStream*, and *Topology.strings* is the simplest. The user specifies a number of strings which will be used as the stream's data items.
-
-
-``` java
- public static void main(String[] args) {
- DirectProvider dp = new DirectProvider();
- Topology top = dp.newTopology();
- TStream<String> helloStream = top.strings("Hello Quarks!");
- }
-```
-
-The *helloStream* stream is created, and the "Hello Quarks!" string will be sent as its single data item.
-## Printing to Output
-*TStream.print* can be used to print the data items of a stream to standard output by invoking the *toString* method of each data item. In this case the data items are already strings, but in principle *TStream.print* can be called on any stream, regardless of the datatype carried by the stream.
-
-``` java
- public static void main(String[] args) {
- DirectProvider dp = new DirectProvider();
- Topology top = dp.newTopology();
- TStream<String> helloStream = top.strings("Hello Quarks!");
- helloStream.print();
- }
-```
-
-## Submitting the Application
-The only remaining step is to submit the application, which is performed my the *DirectProvider*. Submitting a Quarks application initializes the threads which execute the *Topology*, and begins processing its data sources.
-
-``` java
- public static void main(String[] args) {
- DirectProvider dp = new DirectProvider();
- Topology top = dp.newTopology();
- TStream<String> helloStream = top.strings("Hello Quarks!");
- helloStream.print();
- dp.submit(top);
- }
-```
-
-After running the application, the output is "Hello Quarks!":
-
-```
-Hello Quarks!
-```
-
-
-
-
http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/blob/2975c8be/site/docs/recipe_source_function.md
----------------------------------------------------------------------
diff --git a/site/docs/recipe_source_function.md b/site/docs/recipe_source_function.md
deleted file mode 100644
index 2fed684..0000000
--- a/site/docs/recipe_source_function.md
+++ /dev/null
@@ -1,86 +0,0 @@
----
-title: Recipe 2. Writing a Source Function
----
-In the previous [Hello Quarks!](recipe_hello_quarks) example, we create a data source which only generates a single Java String and prints it to output. Yet Quarks sources support the ability generate any data type as a source, not just primitive Java types such as Strings. Moreover, because the user supplies the code which generates the data, the user has complete flexibility for *how* the data is generated. This recipe demonstrates how a user could write such a custom data source.
-
-## Custom Source: Reading the Lines of a Web Page
-{{site.data.alerts.note}} Quarks' API provides convenience methods for performing HTTP requests. For the sake of example we are writing a HTTP data source manually, but in principle there are easier methods. {{site.data.alerts.end}}
-
-One example of a custom data source could be retrieving the contents of a web page and printing each line to output. For example, the user could be querying the Yahoo Finance website for the most recent stock price data of Bank of America, Cabot Oil & Gas, and Freeport-McMoRan Inc:
-
-``` java
- public static void main(String[] args) throws Exception {
- DirectProvider dp = new DirectProvider();
- Topology top = dp.newTopology();
-
- final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
- }
-```
-
-Given the correctly formatted URL to request the data, we can use the *Topology.source* method to generate each line of the page as a data item on the stream. *Topology.source* takes a Java Supplier that returns an Iterable. The supplier is invoked once, and the items returned from the Iterable are used as the stream's data items. For example, the following *queryWebsite* method returns a supplier which queries a URL and returns an Iterable of its contents:
-
-``` java
- private static Supplier<Iterable<String> > queryWebsite(URL url) throws Exception{
- return () -> {
- List<String> lines = new LinkedList<>();
- try {
- InputStream is = url.openStream();
- BufferedReader br = new BufferedReader(
- new InputStreamReader(is));
-
- for(String s = br.readLine(); s != null; s = br.readLine())
- lines.add(s);
-
- } catch (Exception e) {
- e.printStackTrace();
- }
- return lines;
- };
- }
-```
-
- When invoking *Topology.source*, we can use *queryWebsite* to return the required supplier, passing in the URL.
-
- ``` java
- public static void main(String[] args) throws Exception {
- DirectProvider dp = new DirectProvider();
- Topology top = dp.newTopology();
-
- final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
-
- TStream<String> linesOfWebsite = top.source(queryWebsite(url));
-}
- ```
-
- Source methods such as *Topology.source* and *Topology.strings* return a *TStream*. If we print the *linesOfWebsite* stream to standard output and run the application, we can see that it correctly generates the data and feeds it into the Quarks runtime:
-
-Output:
-
-```java
-"BAC","Bank of America Corporation Com",13.150,13.140,"12:00pm - <b>13.145</b>"
-"COG","Cabot Oil & Gas Corporation Com",21.6800,21.6700,"12:00pm - <b>21.6775</b>"
-"FCX","Freeport-McMoRan, Inc. Common S",8.8200,8.8100,"12:00pm - <b>8.8035</b>"
-```
-
-## Polling source: reading data periodically
-A much more common scenario for a developer is the periodic generation of data from a source operator -- a data source may need to be polled every 5 seconds, 3 hours, or any time frame. To this end, *Topology* exposes the *poll* method which can be used to call a function at the frequency of the user's choosing. For example, a user might want to query Yahoo Finance every two seconds to retrieve the most up to date ticker price for a stock:
-
-```java
- public static void main(String[] args) throws Exception {
- DirectProvider dp = new DirectProvider();
- Topology top = dp.newTopology();
-
- final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
-
- TStream<Iterable<String>> source = top.poll(queryWebsite(url), 2, TimeUnit.SECONDS);
- source.print();
-
- dp.submit(top);
- }
-```
-
-**Output:**
-<br>
-<img src="images/pollingSource.gif">
-
-It's important to note that calls to *DirectProvider.submit* are non-blocking; the main thread will exit, and the threads executing the topology will continue to run. (Also, to see changing stock prices, the above example needs to be run during open trading hours. Otherwise, it will simple return the same results every time the website is polled).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/blob/2975c8be/site/recipes/images/pollingSource.gif
----------------------------------------------------------------------
diff --git a/site/recipes/images/pollingSource.gif b/site/recipes/images/pollingSource.gif
new file mode 100644
index 0000000..bc979c6
Binary files /dev/null and b/site/recipes/images/pollingSource.gif differ
http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/blob/2975c8be/site/recipes/recipe_hello_quarks.md
----------------------------------------------------------------------
diff --git a/site/recipes/recipe_hello_quarks.md b/site/recipes/recipe_hello_quarks.md
new file mode 100644
index 0000000..d6d8aea
--- /dev/null
+++ b/site/recipes/recipe_hello_quarks.md
@@ -0,0 +1,62 @@
+---
+title: Recipe 1. Hello Quarks!
+---
+
+Quarks' pure Java implementation is a powerful feature which allows it to be run on the majority of JVM-compatible systems. It also has the added benefit of enabling the developer to develop applications entirely within the Eclipse and Intellij ecosystems. For the purposes of this (and future) recipes, it will be assumed that the developer is using Eclipse. To begin the Hello World recipe, create a new project and import the necessary libraries as outlined in the [getting started guide](quarks-getting-started). Next, write the following template application:
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ }
+```
+
+The *DirectProvider* is an object which allows the user to submit and run the final application. It also creates the *Topology* object, which gives the developer the ability to define a stream of strings.
+
+## Using Topology.strings
+The primary abstraction in Quarks is the `TStream`. A *TStream* represents the flow of data in a Quarks application; for example, the periodic floating point readings from a temperature sensor. The data items which are sent through a `TStream` are Java objects -- in the "Hello Quarks!" example, we are sending a single java.lang.String. There are a number of ways to create a `TStream`, and `Topology.strings` is the simplest. The user specifies a number of strings which will be used as the stream's data items.
+
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ TStream<String> helloStream = top.strings("Hello Quarks!");
+ }
+```
+
+The `helloStream` stream is created, and the "Hello Quarks!" string will be sent as its single data item.
+## Printing to Output
+`TStream.print` can be used to print the data items of a stream to standard output by invoking the `toString` method of each data item. In this case the data items are already strings, but in principle `TStream.print` can be called on any stream, regardless of the datatype carried by the stream.
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ TStream<String> helloStream = top.strings("Hello Quarks!");
+ helloStream.print();
+ }
+```
+
+## Submitting the Application
+The only remaining step is to submit the application, which is performed by the `DirectProvider`. Submitting a Quarks application initializes the threads which execute the `Topology`, and begins processing its data sources.
+
+``` java
+ public static void main(String[] args) {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+ TStream<String> helloStream = top.strings("Hello Quarks!");
+ helloStream.print();
+ dp.submit(top);
+ }
+```
+
+After running the application, the output is "Hello Quarks!":
+
+```
+Hello Quarks!
+```
+
+
+
+
http://git-wip-us.apache.org/repos/asf/incubator-quarks-website/blob/2975c8be/site/recipes/recipe_source_function.md
----------------------------------------------------------------------
diff --git a/site/recipes/recipe_source_function.md b/site/recipes/recipe_source_function.md
new file mode 100644
index 0000000..18ee5ae
--- /dev/null
+++ b/site/recipes/recipe_source_function.md
@@ -0,0 +1,86 @@
+---
+title: Recipe 2. Writing a Source Function
+---
+In the previous [Hello Quarks!](recipe_hello_quarks) example, we create a data source which only generates a single Java String and prints it to output. Yet Quarks sources support the ability generate any data type as a source, not just Java types such as Strings and Doubles. Moreover, because the user supplies the code which generates the data, the user has complete flexibility for *how* the data is generated. This recipe demonstrates how a user could write such a custom data source.
+
+## Custom Source: Reading the Lines of a Web Page
+{{site.data.alerts.note}} Quarks' API provides convenience methods for performing HTTP requests. For the sake of example we are writing a HTTP data source manually, but in principle there are easier methods. {{site.data.alerts.end}}
+
+One example of a custom data source could be retrieving the contents of a web page and printing each line to output. For example, the user could be querying the Yahoo Finance website for the most recent stock price data of Bank of America, Cabot Oil & Gas, and Freeport-McMoRan Inc:
+
+``` java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+ }
+```
+
+Given the correctly formatted URL to request the data, we can use the *Topology.source* method to generate each line of the page as a data item on the stream. `Topology.source` takes a Java Supplier that returns an Iterable. The supplier is invoked once, and the items returned from the Iterable are used as the stream's data items. For example, the following `queryWebsite` method returns a supplier which queries a URL and returns an Iterable of its contents:
+
+``` java
+ private static Supplier<Iterable<String> > queryWebsite(URL url) throws Exception{
+ return () -> {
+ List<String> lines = new LinkedList<>();
+ try {
+ InputStream is = url.openStream();
+ BufferedReader br = new BufferedReader(
+ new InputStreamReader(is));
+
+ for(String s = br.readLine(); s != null; s = br.readLine())
+ lines.add(s);
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return lines;
+ };
+ }
+```
+
+ When invoking `Topology.source`, we can use `queryWebsite` to return the required supplier, passing in the URL.
+
+ ``` java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+
+ TStream<String> linesOfWebsite = top.source(queryWebsite(url));
+ }
+ ```
+
+ Source methods such as `Topology.source` and `Topology.strings` return a `TStream`. If we print the `linesOfWebsite` stream to standard output and run the application, we can see that it correctly generates the data and feeds it into the Quarks runtime:
+
+Output:
+
+```java
+"BAC","Bank of America Corporation Com",13.150,13.140,"12:00pm - <b>13.145</b>"
+"COG","Cabot Oil & Gas Corporation Com",21.6800,21.6700,"12:00pm - <b>21.6775</b>"
+"FCX","Freeport-McMoRan, Inc. Common S",8.8200,8.8100,"12:00pm - <b>8.8035</b>"
+```
+
+## Polling source: reading data periodically
+A much more common scenario for a developer is the periodic generation of data from a source operator -- a data source may need to be polled every 5 seconds, 3 hours, or any time frame. To this end, `Topology` exposes the `poll` method which can be used to call a function at the frequency of the user's choosing. For example, a user might want to query Yahoo Finance every two seconds to retrieve the most up to date ticker price for a stock:
+
+```java
+ public static void main(String[] args) throws Exception {
+ DirectProvider dp = new DirectProvider();
+ Topology top = dp.newTopology();
+
+ final URL url = new URL("http://finance.yahoo.com/d/quotes.csv?s=BAC+COG+FCX&f=snabl");
+
+ TStream<Iterable<String>> source = top.poll(queryWebsite(url), 2, TimeUnit.SECONDS);
+ source.print();
+
+ dp.submit(top);
+ }
+```
+
+**Output:**
+<br>
+<img src="images/pollingSource.gif">
+
+It's important to note that calls to `DirectProvider.submit` are non-blocking; the main thread will exit, and the threads executing the topology will continue to run. (Also, to see changing stock prices, the above example needs to be run during open trading hours. Otherwise, it will simply return the same results every time the website is polled).
\ No newline at end of file