You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Reza ardeshir rokni (JIRA)" <ji...@apache.org> on 2018/10/10 01:53:00 UTC
[jira] [Updated] (BEAM-2953) Create more advanced Timeseries
processing examples using state API
[ https://issues.apache.org/jira/browse/BEAM-2953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reza ardeshir rokni updated BEAM-2953:
--------------------------------------
Description:
A general set of timeseries transforms that abstract the user from the process of dealing with some of the common problems when dealing with timeseries using BEAM (in stream or batch mode).
BEAM can be used to build out some very interesting pre-processing stages for time series data. Some examples that will be useful:
- Downsampling time series based on simple MIN, MAX, COUNT, SUM, LAST, FIRST
- Creating a value for each downsampled window even if no value has been emitted for the specific key.
- Loading the value of a downsample with the previous value (used in FX with previous close being brought into current open value)
was:
As described in the phase 1 portion of this solution outline:
https://cloud.google.com/solutions/correlating-time-series-dataflow
BEAM can be used to build out some very interesting pre-processing stages for time series data. Some examples that will be useful:
- Downsampling time series based on simple AVG, MIN, MAX
- Creating a value for each time window using generatesequence as a seed
- Loading the value of a downsample with the previous value (used in FX with previous close being brought into current open value)
This will show some concrete examples of keyed state as well as the use of combiners.
The samples can also be used to show how you can create a ordered list of values per key from a unbounded topic which has multiple time series keys.
> Create more advanced Timeseries processing examples using state API
> -------------------------------------------------------------------
>
> Key: BEAM-2953
> URL: https://issues.apache.org/jira/browse/BEAM-2953
> Project: Beam
> Issue Type: Improvement
> Components: examples-java
> Affects Versions: 2.1.0
> Reporter: Reza ardeshir rokni
> Assignee: Reuven Lax
> Priority: Minor
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> A general set of timeseries transforms that abstract the user from the process of dealing with some of the common problems when dealing with timeseries using BEAM (in stream or batch mode).
> BEAM can be used to build out some very interesting pre-processing stages for time series data. Some examples that will be useful:
> - Downsampling time series based on simple MIN, MAX, COUNT, SUM, LAST, FIRST
> - Creating a value for each downsampled window even if no value has been emitted for the specific key.
> - Loading the value of a downsample with the previous value (used in FX with previous close being brought into current open value)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)