You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@edgent.apache.org by queeniema <gi...@git.apache.org> on 2016/03/23 01:55:18 UTC
[GitHub] incubator-quarks-website pull request: [QUARKS-36] Add two new rec...
GitHub user queeniema opened a pull request:
https://github.com/apache/incubator-quarks-website/pull/18
[QUARKS-36] Add two new recipes
- Detecting a Sensor Value Out of Expected Range
- Applying Different Processing Against a Single Stream
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/queeniema/incubator-quarks-website recipes-quarks-36
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-quarks-website/pull/18.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18
----
commit 9e985ab8e8c485584d1825ab8b24ca7243eb855d
Author: Queenie Ma <qu...@gmail.com>
Date: 2016-03-23T00:53:10Z
[QUARKS-36] Add recipes for value_out_of_range and different_processing_against_stream
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: [QUARKS-36] Add two new rec...
Posted by ddebrunner <gi...@git.apache.org>.
Github user ddebrunner commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/18#discussion_r57167622
--- Diff: site/recipes/recipe_different_processing_against_stream.md ---
@@ -0,0 +1,150 @@
+---
+title: Recipe 4. Applying Different Processing Against a Single Stream
+---
+
+In the previous [example](recipe_value_out_of_range), we learned how to filter a stream to obtain the interesting sensor readings and ignore the mundane data. Typically, a user scenario is more involved, where data is processed by executing multiple stream operations sequentially. Consider the following scenario, for example.
--- End diff --
previous "recipe" instead of example?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: [QUARKS-36] Add two new rec...
Posted by queeniema <gi...@git.apache.org>.
Github user queeniema commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/18#discussion_r57226844
--- Diff: site/recipes/recipe_different_processing_against_stream.md ---
@@ -0,0 +1,150 @@
+---
+title: Recipe 4. Applying Different Processing Against a Single Stream
--- End diff --
Ah, thank you for the clarification. I will update the recipe/code to follow what you expected.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: [QUARKS-36] Add two new rec...
Posted by ddebrunner <gi...@git.apache.org>.
Github user ddebrunner commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/18#discussion_r57167388
--- Diff: site/recipes/recipe_different_processing_against_stream.md ---
@@ -0,0 +1,150 @@
+---
+title: Recipe 4. Applying Different Processing Against a Single Stream
--- End diff --
I think this one needs a better title, I was expecting an example of fan-out where the a single stream is fed into different functions, (e.g.
```
stream = ...
stream.filter(...);
stream.map(...);
```
Instead this is an example of pipeline processing, a stream is transformed/filtered into a different stream.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: [QUARKS-36] Add two new rec...
Posted by dlaboss <gi...@git.apache.org>.
Github user dlaboss commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/18#discussion_r57102523
--- Diff: site/recipes/recipe_different_processing_against_stream.md ---
@@ -0,0 +1,150 @@
+---
+title: Recipe 4. Applying Different Processing Against a Single Stream
+---
+
+In the previous [example](recipe_value_out_of_range), we learned how to filter a stream to obtain the interesting sensor readings and ignore the mundane data. Typically, a user scenario is more involved, where data is processed by executing multiple stream operations sequentially. Consider the following scenario, for example.
+
+Suppose a package delivery company would like to monitor the gas mileage of their delivery trucks using embedded sensors. They would like to apply different analytics to the sensor data that can be used to make more informed business decisions. For instance, if a truck is reporting consistently poor mileage readings, the company might want to consider replacing that truck to save on gas costs.
+
+In this instance, we can apply different processing against the stream of mileage readings from the sensor and generate warnings when poor gas mileage is detected.
+
+## Setting up the application
+
+We assume that the environment has been set up following the steps outlined in the [Getting Started Guide](../docs/quarks-getting-started). Let's begin by creating a `DirectProvider` and `Topology`. We choose a `DevelopmentProvider` so that we can view the topology graph using the console URL (refer to the [Application Console](../docs/console) page for a more detailed explanation of this provider). The initial mileage value has also been defined.
+
+```java
+ public class TruckSensor {
+ /**
+ * Hypothetical value for initial gas mileage
+ */
+ static double currentMileage = 10.5;
+
+ public static void main(String[] args) throws Exception {
+
+ DirectProvider dp = new DevelopmentProvider();
+
+ System.out.println(dp.getServices().getService(HttpServer.class).getConsoleUrl());
+
+ Topology top = dp.newTopology("TruckSensor");
+
+ // The rest of the code pieces belong here.
+ }
+ }
+```
+
+## Generating mileage sensor readings
+
+The next step is to simulate a stream of gas mileage readings. In our `main()`, we use the `poll()` method to generate a flow of tuples (readings), where each tuple arrives every second.
+
+```java
+ // Generate a stream of mileage sensor readings.
+ Random r = new Random();
+ TStream<Double> mileage = top.poll(() -> {
+ // Change current mileage by some random amount between -0.4 and 0.4.
+ double newMileage = -0.4 + (0.4 + 0.4) * r.nextDouble() + currentMileage;
+ currentMileage = newMileage;
+ return currentMileage;
+ }, 1, TimeUnit.SECONDS);
+```
+
+## Applying different processing to the stream
+
+The company can now perform analytics on the `mileage` stream to generate poor gas mileage warnings by starting with the original stream and altering it. First, we filter out values that are out of range. Then, we tag the stream with the _mileage_ tag for easier viewing in the console (we add this tag after every step). Next, we truncate the reading to one decimal place. We then use `peek` to print out the current mileage so that we can get a sense of the truck's current state. After, we keep the mileage readings that are considered "poor." Finally, we terminate the stream by printing out warnings for poor gas mileage.
+
+```java
+ DecimalFormat df = new DecimalFormat("#.#");
+
+ // Perform analytics on mileage readings to generate poor gas mileage warnings.
+ mileage = mileage.filter(tuple -> tuple >= 7.0 && tuple <= 14.0);
--- End diff --
Seems like a good opportunity to demonstrate, and promote, the use of the fluent style here? It would remove lots of visual noise from all of the unnecessary `mileage = mileage. ...` . e.g.,
```
mileage.filter(...).tag("mileage")
.modify(...).tag("mileage")
.peek(...).tag("mileage")
.filter(...).tag("mileage")
.sink(...);
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: [QUARKS-36] Add two new rec...
Posted by ddebrunner <gi...@git.apache.org>.
Github user ddebrunner commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/18#discussion_r57166605
--- Diff: site/recipes/recipe_different_processing_against_stream.md ---
@@ -0,0 +1,150 @@
+---
+title: Recipe 4. Applying Different Processing Against a Single Stream
+---
+
+In the previous [example](recipe_value_out_of_range), we learned how to filter a stream to obtain the interesting sensor readings and ignore the mundane data. Typically, a user scenario is more involved, where data is processed by executing multiple stream operations sequentially. Consider the following scenario, for example.
+
+Suppose a package delivery company would like to monitor the gas mileage of their delivery trucks using embedded sensors. They would like to apply different analytics to the sensor data that can be used to make more informed business decisions. For instance, if a truck is reporting consistently poor mileage readings, the company might want to consider replacing that truck to save on gas costs.
+
+In this instance, we can apply different processing against the stream of mileage readings from the sensor and generate warnings when poor gas mileage is detected.
+
+## Setting up the application
+
+We assume that the environment has been set up following the steps outlined in the [Getting Started Guide](../docs/quarks-getting-started). Let's begin by creating a `DirectProvider` and `Topology`. We choose a `DevelopmentProvider` so that we can view the topology graph using the console URL (refer to the [Application Console](../docs/console) page for a more detailed explanation of this provider). The initial mileage value has also been defined.
+
+```java
+ public class TruckSensor {
+ /**
+ * Hypothetical value for initial gas mileage
+ */
+ static double currentMileage = 10.5;
+
+ public static void main(String[] args) throws Exception {
+
+ DirectProvider dp = new DevelopmentProvider();
+
+ System.out.println(dp.getServices().getService(HttpServer.class).getConsoleUrl());
+
+ Topology top = dp.newTopology("TruckSensor");
+
+ // The rest of the code pieces belong here.
+ }
+ }
+```
+
+## Generating mileage sensor readings
+
+The next step is to simulate a stream of gas mileage readings. In our `main()`, we use the `poll()` method to generate a flow of tuples (readings), where each tuple arrives every second.
+
+```java
+ // Generate a stream of mileage sensor readings.
+ Random r = new Random();
+ TStream<Double> mileage = top.poll(() -> {
+ // Change current mileage by some random amount between -0.4 and 0.4.
+ double newMileage = -0.4 + (0.4 + 0.4) * r.nextDouble() + currentMileage;
+ currentMileage = newMileage;
+ return currentMileage;
+ }, 1, TimeUnit.SECONDS);
+```
+
+## Applying different processing to the stream
+
+The company can now perform analytics on the `mileage` stream to generate poor gas mileage warnings by starting with the original stream and altering it. First, we filter out values that are out of range. Then, we tag the stream with the _mileage_ tag for easier viewing in the console (we add this tag after every step). Next, we truncate the reading to one decimal place. We then use `peek` to print out the current mileage so that we can get a sense of the truck's current state. After, we keep the mileage readings that are considered "poor." Finally, we terminate the stream by printing out warnings for poor gas mileage.
+
+```java
+ DecimalFormat df = new DecimalFormat("#.#");
+
+ // Perform analytics on mileage readings to generate poor gas mileage warnings.
+ mileage = mileage.filter(tuple -> tuple >= 7.0 && tuple <= 14.0);
+ mileage = mileage.tag("mileage");
+ mileage = mileage.modify(tuple -> Double.valueOf(df.format(tuple)));
+ mileage = mileage.tag("mileage");
+ mileage = mileage.peek(tuple -> System.out.println("Mileage: " + tuple + " mpg"));
+ mileage = mileage.tag("mileage");
--- End diff --
There's no need for the tag() after the peek() as line 61 already added the mileage tag.
Note peek returns 'this' so local variable mileage has the same reference for the assignment on lines 60,61,62
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: [QUARKS-36] Add two new rec...
Posted by queeniema <gi...@git.apache.org>.
Github user queeniema commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/18#discussion_r57226356
--- Diff: site/recipes/recipe_different_processing_against_stream.md ---
@@ -0,0 +1,150 @@
+---
+title: Recipe 4. Applying Different Processing Against a Single Stream
+---
+
+In the previous [example](recipe_value_out_of_range), we learned how to filter a stream to obtain the interesting sensor readings and ignore the mundane data. Typically, a user scenario is more involved, where data is processed by executing multiple stream operations sequentially. Consider the following scenario, for example.
+
+Suppose a package delivery company would like to monitor the gas mileage of their delivery trucks using embedded sensors. They would like to apply different analytics to the sensor data that can be used to make more informed business decisions. For instance, if a truck is reporting consistently poor mileage readings, the company might want to consider replacing that truck to save on gas costs.
+
+In this instance, we can apply different processing against the stream of mileage readings from the sensor and generate warnings when poor gas mileage is detected.
+
+## Setting up the application
+
+We assume that the environment has been set up following the steps outlined in the [Getting Started Guide](../docs/quarks-getting-started). Let's begin by creating a `DirectProvider` and `Topology`. We choose a `DevelopmentProvider` so that we can view the topology graph using the console URL (refer to the [Application Console](../docs/console) page for a more detailed explanation of this provider). The initial mileage value has also been defined.
+
+```java
+ public class TruckSensor {
+ /**
+ * Hypothetical value for initial gas mileage
+ */
+ static double currentMileage = 10.5;
+
+ public static void main(String[] args) throws Exception {
+
+ DirectProvider dp = new DevelopmentProvider();
+
+ System.out.println(dp.getServices().getService(HttpServer.class).getConsoleUrl());
+
+ Topology top = dp.newTopology("TruckSensor");
+
+ // The rest of the code pieces belong here.
+ }
+ }
+```
+
+## Generating mileage sensor readings
+
+The next step is to simulate a stream of gas mileage readings. In our `main()`, we use the `poll()` method to generate a flow of tuples (readings), where each tuple arrives every second.
+
+```java
+ // Generate a stream of mileage sensor readings.
+ Random r = new Random();
+ TStream<Double> mileage = top.poll(() -> {
+ // Change current mileage by some random amount between -0.4 and 0.4.
+ double newMileage = -0.4 + (0.4 + 0.4) * r.nextDouble() + currentMileage;
+ currentMileage = newMileage;
+ return currentMileage;
+ }, 1, TimeUnit.SECONDS);
+```
+
+## Applying different processing to the stream
+
+The company can now perform analytics on the `mileage` stream to generate poor gas mileage warnings by starting with the original stream and altering it. First, we filter out values that are out of range. Then, we tag the stream with the _mileage_ tag for easier viewing in the console (we add this tag after every step). Next, we truncate the reading to one decimal place. We then use `peek` to print out the current mileage so that we can get a sense of the truck's current state. After, we keep the mileage readings that are considered "poor." Finally, we terminate the stream by printing out warnings for poor gas mileage.
+
+```java
+ DecimalFormat df = new DecimalFormat("#.#");
+
+ // Perform analytics on mileage readings to generate poor gas mileage warnings.
+ mileage = mileage.filter(tuple -> tuple >= 7.0 && tuple <= 14.0);
--- End diff --
I thought that performing the steps separately would make it easier to follow, but I agree that it would be cleaner to use the fluent style. I will do that for this and future recipes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: [QUARKS-36] Add two new rec...
Posted by queeniema <gi...@git.apache.org>.
Github user queeniema commented on a diff in the pull request:
https://github.com/apache/incubator-quarks-website/pull/18#discussion_r57226855
--- Diff: site/recipes/recipe_different_processing_against_stream.md ---
@@ -0,0 +1,150 @@
+---
+title: Recipe 4. Applying Different Processing Against a Single Stream
+---
+
+In the previous [example](recipe_value_out_of_range), we learned how to filter a stream to obtain the interesting sensor readings and ignore the mundane data. Typically, a user scenario is more involved, where data is processed by executing multiple stream operations sequentially. Consider the following scenario, for example.
--- End diff --
Yes, that's what I meant. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quarks-website pull request: [QUARKS-36] Add two new rec...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-quarks-website/pull/18
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---