You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2023/01/02 16:36:33 UTC
[GitHub] [flink-ml] zhipeng93 opened a new pull request, #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
zhipeng93 opened a new pull request, #196:
URL: https://github.com/apache/flink-ml/pull/196
## What is the purpose of the change
- Add Transformer and Estimator for OnlineStandardScaler
## Brief change log
- Added Transformer and Estimator for OnlineStandardScaler (java/python source/test/example and docs).
- Added two metrics for exposing ML model attributes (i.e., `timestamp` and `version` of model data).
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1094460626
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(MLMetrics.ML_MODEL_GROUP);
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() <= modelTimeStamp + maxAllowedModelDelayMs
Review Comment:
Thanks for the fix. LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1066574806
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScaler.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.ProcessAllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * An Estimator which implements the online standard scaling algorithm, which is the online version
+ * of {@link StandardScaler}.
+ *
+ * <p>OnlineStandardScaler splits the input data by the user-specified window strategy (i.e., {@link
+ * org.apache.flink.ml.common.param.HasWindows}). For each window, it computes the mean and standard
+ * deviation using the data seen so far (i.e., not only the data in the current window, but also the
+ * history data). The model data generated by OnlineStandardScaler is a model stream. There is one
+ * model data for each window.
+ *
+ * <p>During the inference phase (i.e., using {@link OnlineStandardScalerModel} for prediction),
+ * users could output the model version that is used for predicting each data point. Moreover,
+ *
+ * <ul>
+ * <li>When the train data and test data both contains event time, users could specify the maximum
+ * difference between timestamp of the input and model data ({@link
+ * org.apache.flink.ml.common.param.HasMaxAllowedModelDelayMs}), which enforces to use a
+ * relatively fresh model for prediction.
+ * <li>Otherwise, the prediction process always use the current model data for prediction.
+ * </ul>
+ */
+public class OnlineStandardScaler
+ implements Estimator<OnlineStandardScaler, OnlineStandardScalerModel>,
+ OnlineStandardScalerParams<OnlineStandardScaler> {
+
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public OnlineStandardScaler() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public OnlineStandardScalerModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<StandardScalerModelData> modelData =
+ DataStreamUtils.windowAllAndProcess(
+ tEnv.toDataStream(inputs[0]),
+ getWindows(),
+ new ComputeModelDataFunction<>(getInputCol()));
+
+ OnlineStandardScalerModel model =
+ new OnlineStandardScalerModel().setModelData(tEnv.fromDataStream(modelData));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ private static class ComputeModelDataFunction<W extends Window>
+ extends ProcessAllWindowFunction<Row, StandardScalerModelData, W> {
+
+ private final String inputCol;
+
+ public ComputeModelDataFunction(String inputCol) {
+ this.inputCol = inputCol;
+ }
+
+ @Override
+ public void process(
+ ProcessAllWindowFunction<Row, StandardScalerModelData, W>.Context context,
+ Iterable<Row> iterable,
+ Collector<StandardScalerModelData> collector)
+ throws Exception {
+ ListState<DenseVector> sumState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>(
+ "sumState", DenseVectorTypeInfo.INSTANCE));
+ ListState<DenseVector> squaredSumState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>(
+ "squaredSumState", DenseVectorTypeInfo.INSTANCE));
+ ListState<Long> numElementsState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>("numElementsState", Types.LONG));
+ ListState<Long> modelVersionState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>("modelVersionState", Types.LONG));
+ DenseVector sum =
+ OperatorStateUtils.getUniqueElement(sumState, "sumState").orElse(null);
+ DenseVector squaredSum =
+ OperatorStateUtils.getUniqueElement(squaredSumState, "squaredSumState")
+ .orElse(null);
+ long numElements =
+ OperatorStateUtils.getUniqueElement(numElementsState, "numElementsState")
+ .orElse(0L);
+ long modelVersion =
+ OperatorStateUtils.getUniqueElement(modelVersionState, "modelVersionState")
+ .orElse(0L);
+
+ long numElementsBefore = numElements;
+ for (Row element : iterable) {
+ Vector inputVec =
+ ((Vector) Objects.requireNonNull(element.getField(inputCol))).clone();
+ if (numElements == 0) {
Review Comment:
I am afraid we cannot do this --- Puting it inside the loop is to get the size of the vector for initializing `sum` and `squareSum`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1093183274
##########
docs/content/docs/operators/feature/onlinestandardscaler.md:
##########
@@ -0,0 +1,261 @@
+---
+title: "OnlineStandardScaler"
+weight: 1
+type: docs
+aliases:
+- /operators/feature/onlinestandardscaler.html
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## OnlineStandardScaler
+
+An Estimator which implements the online standard scaling algorithm, which
+is the online version of StandardScaler.
+
+OnlineStandardScaler splits the input data by the user-specified window strategy.
+For each window, it computes the mean and standard deviation using the data seen
+so far (i.e., not only the data in the current window, but also the history data).
+The model data generated by OnlineStandardScaler is a model stream.
+There is one model data for each window.
+
+During the inference phase (i.e., using OnlineStandardScalerModel for prediction),
+users could output the model version that is used for predicting each data point.
+Moreover,
+- When the train data and test data both contain event time, users could
+specify the maximum difference between the timestamps of the input and model data,
+which enforces to use a relatively fresh model for prediction.
+- Otherwise, the prediction process always uses the current model data for prediction.
+
+
+### Input Columns
+
+| Param name | Type | Default | Description |
+|:-----------|:-------|:----------|:-----------------------|
+| inputCol | Vector | `"input"` | Features to be scaled. |
+
+### Output Columns
+
+| Param name | Type | Default | Description |
+|:----------------|:-------|:-----------|:--------------------------------------------------------------------------------------|
+| outputCol | Vector | `"output"` | Scaled features. |
+| modelVersionCol | String | `version` | The version (in long format) of the model data that the input data is predicted with. |
+
+### Parameters
+
+Below are the parameters required by `OnlineStandardScalerModel`.
+
+| Key | Default | Type | Required | Description |
+|------------------------|------------|---------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| inputCol | `"input"` | String | no | Input column name. |
+| outputCol | `"output"` | String | no | Output column name. |
+| withMean | `false` | Boolean | no | Whether centers the data with mean before scaling. |
+| withStd | `true` | Boolean | no | Whether scales the data with standard deviation. |
+| modelVersionCol | `null` | String | no | The version (in long format) of the model data that the input data is predicted with. |
+| maxAllowedModelDelayMs | `0L` | Long | no | The maximum difference allowed between the timestamps of the input record and the model data that is used to predict that input record. This param only works when the input contains event time. |
+
+`OnlineStandardScaler` needs parameters above and also below.
+
+| Key | Default | Type | Required | Description |
+|---------|-------------------------------|---------|----------|--------------------------------------------------------------------------------|
+| windows | `GlobalWindows.getInstance()` | Windows | no | Windowing strategy that determines how to create mini-batches from input data. |
+
+
+### Examples
+
+{{< tabs examples >}}
+
+{{< tab "Java">}}
+
+```java
+import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.ml.common.window.EventTimeTumblingWindows;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScaler;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScalerModel;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import java.util.Arrays;
+import java.util.List;
+
+/** Simple program that trains a OnlineStandardScaler model and uses it for feature engineering. */
+public class OnlineStandardScalerExample {
+ public static void main(String[] args) {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ // Generates input data.
+ List<Row> inputData =
+ Arrays.asList(
+ Row.of(0L, Vectors.dense(-2.5, 9, 1)),
+ Row.of(1000L, Vectors.dense(1.4, -5, 1)),
+ Row.of(2000L, Vectors.dense(2, -1, -2)),
+ Row.of(6000L, Vectors.dense(0.7, 3, 1)),
+ Row.of(7000L, Vectors.dense(0, 1, 1)),
+ Row.of(8000L, Vectors.dense(0.5, 0, -2)),
+ Row.of(9000L, Vectors.dense(0.4, 1, 1)),
+ Row.of(10000L, Vectors.dense(0.3, 2, 1)),
+ Row.of(11000L, Vectors.dense(0.5, 1, -2)));
+
+ DataStream<Row> inputStream = env.fromCollection(inputData);
+
+ DataStream<Row> inputStreamWithEventTime =
+ inputStream.assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Row>forMonotonousTimestamps()
+ .withTimestampAssigner(
+ (SerializableTimestampAssigner<Row>)
+ (element, recordTimestamp) ->
+ element.getFieldAs(0)));
+
+ Table inputTable =
+ tEnv.fromDataStream(
+ inputStreamWithEventTime,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build())
+ .as("id", "input");
+
+ // Creates an OnlineStandardScaler object and initializes its parameters.
+ long windowSizeMs = 3000;
+ OnlineStandardScaler onlineStandardScaler =
+ new OnlineStandardScaler()
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(windowSizeMs)))
+ .setModelVersionCol("modelVersionCol");
Review Comment:
Now that we have default value for this parameter, would it be simpler to skip setting it?
It would also be more consistent the pattern of not explicitly setting input/output columns.
Same for `OnlineStandardScalerExample`.
##########
docs/content/docs/operators/feature/onlinestandardscaler.md:
##########
@@ -0,0 +1,261 @@
+---
+title: "OnlineStandardScaler"
+weight: 1
+type: docs
+aliases:
+- /operators/feature/onlinestandardscaler.html
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## OnlineStandardScaler
+
+An Estimator which implements the online standard scaling algorithm, which
+is the online version of StandardScaler.
+
+OnlineStandardScaler splits the input data by the user-specified window strategy.
+For each window, it computes the mean and standard deviation using the data seen
+so far (i.e., not only the data in the current window, but also the history data).
+The model data generated by OnlineStandardScaler is a model stream.
+There is one model data for each window.
+
+During the inference phase (i.e., using OnlineStandardScalerModel for prediction),
+users could output the model version that is used for predicting each data point.
+Moreover,
+- When the train data and test data both contain event time, users could
+specify the maximum difference between the timestamps of the input and model data,
+which enforces to use a relatively fresh model for prediction.
+- Otherwise, the prediction process always uses the current model data for prediction.
+
+
+### Input Columns
+
+| Param name | Type | Default | Description |
+|:-----------|:-------|:----------|:-----------------------|
+| inputCol | Vector | `"input"` | Features to be scaled. |
+
+### Output Columns
+
+| Param name | Type | Default | Description |
+|:----------------|:-------|:-----------|:--------------------------------------------------------------------------------------|
+| outputCol | Vector | `"output"` | Scaled features. |
+| modelVersionCol | String | `version` | The version (in long format) of the model data that the input data is predicted with. |
+
+### Parameters
+
+Below are the parameters required by `OnlineStandardScalerModel`.
+
+| Key | Default | Type | Required | Description |
+|------------------------|------------|---------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| inputCol | `"input"` | String | no | Input column name. |
+| outputCol | `"output"` | String | no | Output column name. |
+| withMean | `false` | Boolean | no | Whether centers the data with mean before scaling. |
+| withStd | `true` | Boolean | no | Whether scales the data with standard deviation. |
+| modelVersionCol | `null` | String | no | The version (in long format) of the model data that the input data is predicted with. |
+| maxAllowedModelDelayMs | `0L` | Long | no | The maximum difference allowed between the timestamps of the input record and the model data that is used to predict that input record. This param only works when the input contains event time. |
+
+`OnlineStandardScaler` needs parameters above and also below.
+
+| Key | Default | Type | Required | Description |
+|---------|-------------------------------|---------|----------|--------------------------------------------------------------------------------|
+| windows | `GlobalWindows.getInstance()` | Windows | no | Windowing strategy that determines how to create mini-batches from input data. |
+
+
+### Examples
+
+{{< tabs examples >}}
+
+{{< tab "Java">}}
+
+```java
+import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.ml.common.window.EventTimeTumblingWindows;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScaler;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScalerModel;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import java.util.Arrays;
+import java.util.List;
+
+/** Simple program that trains a OnlineStandardScaler model and uses it for feature engineering. */
+public class OnlineStandardScalerExample {
+ public static void main(String[] args) {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ // Generates input data.
+ List<Row> inputData =
+ Arrays.asList(
+ Row.of(0L, Vectors.dense(-2.5, 9, 1)),
+ Row.of(1000L, Vectors.dense(1.4, -5, 1)),
+ Row.of(2000L, Vectors.dense(2, -1, -2)),
+ Row.of(6000L, Vectors.dense(0.7, 3, 1)),
+ Row.of(7000L, Vectors.dense(0, 1, 1)),
+ Row.of(8000L, Vectors.dense(0.5, 0, -2)),
+ Row.of(9000L, Vectors.dense(0.4, 1, 1)),
+ Row.of(10000L, Vectors.dense(0.3, 2, 1)),
+ Row.of(11000L, Vectors.dense(0.5, 1, -2)));
+
+ DataStream<Row> inputStream = env.fromCollection(inputData);
+
+ DataStream<Row> inputStreamWithEventTime =
+ inputStream.assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Row>forMonotonousTimestamps()
+ .withTimestampAssigner(
+ (SerializableTimestampAssigner<Row>)
+ (element, recordTimestamp) ->
+ element.getFieldAs(0)));
+
+ Table inputTable =
+ tEnv.fromDataStream(
+ inputStreamWithEventTime,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build())
+ .as("id", "input");
+
+ // Creates an OnlineStandardScaler object and initializes its parameters.
+ long windowSizeMs = 3000;
+ OnlineStandardScaler onlineStandardScaler =
+ new OnlineStandardScaler()
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(windowSizeMs)))
+ .setModelVersionCol("modelVersionCol");
+
+ // Trains the OnlineStandardScaler Model.
+ OnlineStandardScalerModel model = onlineStandardScaler.fit(inputTable);
+
+ // Uses the OnlineStandardScaler Model for predictions.
+ Table outputTable = model.transform(inputTable)[0];
+
+ // Extracts and displays the results.
+ for (CloseableIterator<Row> it = outputTable.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ DenseVector inputValue = (DenseVector) row.getField(onlineStandardScaler.getInputCol());
+ DenseVector outputValue =
+ (DenseVector) row.getField(onlineStandardScaler.getOutputCol());
+ long modelVersion = row.getFieldAs(onlineStandardScaler.getModelVersionCol());
+ System.out.printf(
+ "Input Value: %s\tOutput Value: %s\tModel Version: %s\n",
+ inputValue, outputValue, modelVersion);
+ }
+ }
+}
+
+```
+
+{{< /tab>}}
+
+{{< tab "Python">}}
+
+```python
+# Simple program that trains an OnlineStandardScaler model and uses it for feature
+# engineering.
+
+from pyflink.common import Types
+from pyflink.common.time import Time, Instant
+from pyflink.java_gateway import get_gateway
+from pyflink.table import Schema
+from pyflink.datastream import StreamExecutionEnvironment
+from pyflink.table import StreamTableEnvironment
+from pyflink.table.expressions import col
+
+from pyflink.ml.core.linalg import Vectors, DenseVectorTypeInfo
+from pyflink.ml.lib.feature.onlinestandardscaler import OnlineStandardScaler
+from pyflink.ml.core.windows import EventTimeTumblingWindows
+
+# Creates a new StreamExecutionEnvironment.
+env = StreamExecutionEnvironment.get_execution_environment()
+
+# Creates a StreamTableEnvironment.
+t_env = StreamTableEnvironment.create(env)
+
+# Generates input data.
+dense_vector_serializer = get_gateway().jvm.org.apache.flink.table.types.logical.RawType(
+ get_gateway().jvm.org.apache.flink.ml.linalg.DenseVector(0).getClass(),
+ get_gateway().jvm.org.apache.flink.ml.linalg.typeinfo.DenseVectorSerializer()
+).getSerializerString()
+
+schema = Schema.new_builder()
+ .column("ts", "TIMESTAMP_LTZ(3)")
+ .column("input", "RAW('org.apache.flink.ml.linalg.DenseVector', '{serializer}')"
+ .format(serializer=dense_vector_serializer))
+ .watermark("ts", "ts - INTERVAL '1' SECOND")
+ .build()
+
+input_data = t_env.from_data_stream(
+ env.from_collection([
+ (Instant.of_epoch_milli(0), Vectors.dense(-2.5, 9, 1),),
+ (Instant.of_epoch_milli(1000), Vectors.dense(1.4, -5, 1),),
+ (Instant.of_epoch_milli(2000), Vectors.dense(2, -1, -2),),
+ (Instant.of_epoch_milli(6000), Vectors.dense(0.7, 3, 1),),
+ (Instant.of_epoch_milli(7000), Vectors.dense(0, 1, 1),),
+ (Instant.of_epoch_milli(8000), Vectors.dense(0.5, 0, -2),),
+ (Instant.of_epoch_milli(9000), Vectors.dense(0.4, 1, 1),),
+ (Instant.of_epoch_milli(10000), Vectors.dense(0.3, 2, 1),),
+ (Instant.of_epoch_milli(11000), Vectors.dense(0.5, 1, -2),)
+ ],
+ type_info=Types.ROW_NAMED(
+ ['ts', 'input'],
+ [Types.INSTANT(), DenseVectorTypeInfo()])),
+ schema)
+
+# Creates an online standard-scaler object and initialize its parameters.
+standard_scaler = OnlineStandardScaler()
+ .set_windows(EventTimeTumblingWindows.of(Time.milliseconds(3000)))
+ .set_model_version_col('model_version_col')
Review Comment:
Remove this?
##########
flink-ml-python/pyflink/ml/lib/feature/tests/test_onlinestandardscaler.py:
##########
@@ -0,0 +1,216 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from pyflink.common import Types
+from pyflink.common.time import Time, Instant
+from pyflink.java_gateway import get_gateway
+from pyflink.table import Schema
+from pyflink.table.types import DataTypes
+from pyflink.table.expressions import col
+
+from pyflink.ml.core.linalg import Vectors, DenseVectorTypeInfo
+from pyflink.ml.lib.feature.onlinestandardscaler import OnlineStandardScaler, \
+ OnlineStandardScalerModel
+from pyflink.ml.tests.test_utils import PyFlinkMLTestCase, update_existing_params
+from pyflink.ml.core.windows import GlobalWindows, EventTimeTumblingWindows
+
+
+class OnlineStandardScalerTest(PyFlinkMLTestCase):
+ def setUp(self):
+ super(OnlineStandardScalerTest, self).setUp()
+
+ dense_vector_serializer = get_gateway().jvm.org.apache.flink.table.types.logical.RawType(
+ get_gateway().jvm.org.apache.flink.ml.linalg.DenseVector(0).getClass(),
+ get_gateway().jvm.org.apache.flink.ml.linalg.typeinfo.DenseVectorSerializer()
+ ).getSerializerString()
+
+ schema = Schema.new_builder() \
+ .column("ts_in_long", DataTypes.BIGINT()) \
+ .column("ts", "TIMESTAMP_LTZ(3)") \
+ .column("input", "RAW('org.apache.flink.ml.linalg.DenseVector', '{serializer}')"
+ .format(serializer=dense_vector_serializer)) \
+ .watermark("ts", "ts - INTERVAL '1' SECOND") \
+ .build()
+
+ self.input_table = self.t_env.from_data_stream(
+ self.env.from_collection([
+ (0, Instant.of_epoch_milli(0), Vectors.dense(-2.5, 9, 1),),
+ (1000, Instant.of_epoch_milli(1000), Vectors.dense(1.4, -5, 1),),
+ (2000, Instant.of_epoch_milli(2000), Vectors.dense(2, -1, -2),),
+ (6000, Instant.of_epoch_milli(6000), Vectors.dense(0.7, 3, 1),),
+ (7000, Instant.of_epoch_milli(7000), Vectors.dense(0, 1, 1),),
+ (8000, Instant.of_epoch_milli(8000), Vectors.dense(0.5, 0, -2),),
+ (9000, Instant.of_epoch_milli(9000), Vectors.dense(0.4, 1, 1),),
+ (10000, Instant.of_epoch_milli(10000), Vectors.dense(0.3, 2, 1),),
+ (11000, Instant.of_epoch_milli(11000), Vectors.dense(0.5, 1, -2),)
+ ],
+ type_info=Types.ROW_NAMED(
+ ['ts_in_long', 'ts', 'input'],
+ [Types.LONG(), Types.INSTANT(), DenseVectorTypeInfo()])),
+ schema)
+
+ self.window_size_ms = 3000
+
+ self.expected_model_data = [
+ [
+ Vectors.dense(0.3, 1, 0),
+ Vectors.dense(2.4433583, 7.2111026, 1.7320508),
+ 0,
+ 2999
+ ],
+ [
+ Vectors.dense(0.35, 1.1666667, 0),
+ Vectors.dense(1.5630099, 4.6654760, 1.5491933),
+ 1,
+ 8999
+ ],
+ [
+ Vectors.dense(0.3666667, 1.2222222, 0),
+ Vectors.dense(1.2369316, 3.7006005, 1.5),
+ 2,
+ 11999
+ ]
+ ]
+
+ self.tolerance = 1e-7
+
+ def test_param(self):
+ standard_scaler = OnlineStandardScaler()
+
+ self.assertEqual('input', standard_scaler.input_col)
+ self.assertEqual(False, standard_scaler.with_mean)
+ self.assertEqual(True, standard_scaler.with_std)
+ self.assertEqual('output', standard_scaler.output_col)
+ self.assertEqual('version', standard_scaler.model_version_col)
+ self.assertEqual(GlobalWindows(), standard_scaler.windows)
+ self.assertEqual(0, standard_scaler.max_allowed_model_delay_ms)
+
+ standard_scaler.set_input_col('test_input') \
+ .set_with_mean(True) \
+ .set_with_std(False) \
+ .set_output_col('test_output') \
+ .set_model_version_col('model_version_col') \
Review Comment:
Change the value to something that is not the default value?
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(
+ MLMetrics.ML_MODEL_GROUP,
+ OnlineStandardScalerModel.class.getSimpleName());
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() - maxAllowedModelDelayMs <= modelTimeStamp
+ && mean != null) {
Review Comment:
Is this line needed to make sure that we have received model data before processing data?
Do we still need it if set the default value of modelTimeStamp to `Long.MIN_VALUE`?
##########
flink-ml-examples/src/main/java/org/apache/flink/ml/examples/feature/OnlineStandardScalerExample.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.examples.feature;
+
+import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.ml.common.window.EventTimeTumblingWindows;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScaler;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScalerModel;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import java.util.Arrays;
+import java.util.List;
+
+/** Simple program that trains a OnlineStandardScaler model and uses it for feature engineering. */
+public class OnlineStandardScalerExample {
+ public static void main(String[] args) {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ // Generates input data.
+ List<Row> inputData =
+ Arrays.asList(
+ Row.of(0L, Vectors.dense(-2.5, 9, 1)),
+ Row.of(1000L, Vectors.dense(1.4, -5, 1)),
+ Row.of(2000L, Vectors.dense(2, -1, -2)),
+ Row.of(6000L, Vectors.dense(0.7, 3, 1)),
+ Row.of(7000L, Vectors.dense(0, 1, 1)),
+ Row.of(8000L, Vectors.dense(0.5, 0, -2)),
+ Row.of(9000L, Vectors.dense(0.4, 1, 1)),
+ Row.of(10000L, Vectors.dense(0.3, 2, 1)),
+ Row.of(11000L, Vectors.dense(0.5, 1, -2)));
+
+ DataStream<Row> inputStream = env.fromCollection(inputData);
+
+ DataStream<Row> inputStreamWithEventTime =
+ inputStream.assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Row>forMonotonousTimestamps()
+ .withTimestampAssigner(
+ (SerializableTimestampAssigner<Row>)
+ (element, recordTimestamp) ->
+ element.getFieldAs(0)));
+
+ Table inputTable =
+ tEnv.fromDataStream(
+ inputStreamWithEventTime,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build())
+ .as("id", "input");
+
+ // Creates an OnlineStandardScaler object and initializes its parameters.
+ long windowSizeMs = 3000;
+ OnlineStandardScaler onlineStandardScaler =
+ new OnlineStandardScaler()
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(windowSizeMs)))
+ .setModelVersionCol("modelVersionCol");
Review Comment:
Remove this line?
##########
flink-ml-python/pyflink/examples/ml/feature/online_standardscaler_example.py:
##########
@@ -0,0 +1,91 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+# Simple program that trains an OnlineStandardScaler model and uses it for feature
+# engineering.
+
+from pyflink.common import Types
+from pyflink.common.time import Time, Instant
+from pyflink.java_gateway import get_gateway
+from pyflink.table import Schema
+from pyflink.datastream import StreamExecutionEnvironment
+from pyflink.table import StreamTableEnvironment
+from pyflink.table.expressions import col
+
+from pyflink.ml.core.linalg import Vectors, DenseVectorTypeInfo
+from pyflink.ml.lib.feature.onlinestandardscaler import OnlineStandardScaler
+from pyflink.ml.core.windows import EventTimeTumblingWindows
+
+# Creates a new StreamExecutionEnvironment.
+env = StreamExecutionEnvironment.get_execution_environment()
+
+# Creates a StreamTableEnvironment.
+t_env = StreamTableEnvironment.create(env)
+
+# Generates input data.
+dense_vector_serializer = get_gateway().jvm.org.apache.flink.table.types.logical.RawType(
+ get_gateway().jvm.org.apache.flink.ml.linalg.DenseVector(0).getClass(),
+ get_gateway().jvm.org.apache.flink.ml.linalg.typeinfo.DenseVectorSerializer()
+).getSerializerString()
+
+schema = Schema.new_builder() \
+ .column("ts", "TIMESTAMP_LTZ(3)") \
+ .column("input", "RAW('org.apache.flink.ml.linalg.DenseVector', '{serializer}')"
+ .format(serializer=dense_vector_serializer)) \
+ .watermark("ts", "ts - INTERVAL '1' SECOND") \
+ .build()
+
+input_data = t_env.from_data_stream(
+ env.from_collection([
+ (Instant.of_epoch_milli(0), Vectors.dense(-2.5, 9, 1),),
+ (Instant.of_epoch_milli(1000), Vectors.dense(1.4, -5, 1),),
+ (Instant.of_epoch_milli(2000), Vectors.dense(2, -1, -2),),
+ (Instant.of_epoch_milli(6000), Vectors.dense(0.7, 3, 1),),
+ (Instant.of_epoch_milli(7000), Vectors.dense(0, 1, 1),),
+ (Instant.of_epoch_milli(8000), Vectors.dense(0.5, 0, -2),),
+ (Instant.of_epoch_milli(9000), Vectors.dense(0.4, 1, 1),),
+ (Instant.of_epoch_milli(10000), Vectors.dense(0.3, 2, 1),),
+ (Instant.of_epoch_milli(11000), Vectors.dense(0.5, 1, -2),)
+ ],
+ type_info=Types.ROW_NAMED(
+ ['ts', 'input'],
+ [Types.INSTANT(), DenseVectorTypeInfo()])),
+ schema)
+
+# Creates an online standard-scaler object and initialize its parameters.
+standard_scaler = OnlineStandardScaler() \
+ .set_windows(EventTimeTumblingWindows.of(Time.milliseconds(3000))) \
+ .set_model_version_col('model_version_col') \
Review Comment:
Remove this line?
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasModelVersionCol.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.StringParam;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared model version column param. */
+public interface HasModelVersionCol<T> extends WithParams<T> {
+ Param<String> MODEL_VERSION_COL =
+ new StringParam(
+ "modelVersionCol",
+ "The version (in long format) of the model data that the input data is predicted with.",
Review Comment:
The doc seems confusing. The semantics of this doc is actually "The version column name", not the version of the model.
How about using the following doc:
The name of the column which contains the version of the model data that the input data is predicted with. The version should be a 64-bit integer.
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(
+ MLMetrics.ML_MODEL_GROUP,
+ OnlineStandardScalerModel.class.getSimpleName());
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() - maxAllowedModelDelayMs <= modelTimeStamp
Review Comment:
What is the value of `modelTimeStamp` if `processElement2()` has not been invoked? Should we explicitly give it a default value?
In general it is a good practice to make sure all variables are initialized in the constructor (or `open()` in this case). Maybe do it for other variables as well.
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/OnlineStandardScalerTest.java:
##########
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature;
+
+import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.common.window.CountTumblingWindows;
+import org.apache.flink.ml.common.window.EventTimeTumblingWindows;
+import org.apache.flink.ml.common.window.GlobalWindows;
+import org.apache.flink.ml.common.window.ProcessingTimeTumblingWindows;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScaler;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScalerModel;
+import org.apache.flink.ml.feature.standardscaler.StandardScalerModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.TestUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.test.util.AbstractTestBase;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** Tests {@link OnlineStandardScaler} and {@link OnlineStandardScalerModel}. */
+public class OnlineStandardScalerTest extends AbstractTestBase {
+ @Rule public final TemporaryFolder tempFolder = new TemporaryFolder();
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private final List<Row> inputData =
+ Arrays.asList(
+ Row.of(0L, Vectors.dense(-2.5, 9, 1)),
+ Row.of(1000L, Vectors.dense(1.4, -5, 1)),
+ Row.of(2000L, Vectors.dense(2, -1, -2)),
+ Row.of(6000L, Vectors.dense(0.7, 3, 1)),
+ Row.of(7000L, Vectors.dense(0, 1, 1)),
+ Row.of(8000L, Vectors.dense(0.5, 0, -2)),
+ Row.of(9000L, Vectors.dense(0.4, 1, 1)),
+ Row.of(10000L, Vectors.dense(0.3, 2, 1)),
+ Row.of(11000L, Vectors.dense(0.5, 1, -2)));
+
+ private final List<StandardScalerModelData> expectedModelData =
+ Arrays.asList(
+ new StandardScalerModelData(
+ Vectors.dense(0.3, 1, 0),
+ Vectors.dense(2.4433583, 7.2111026, 1.7320508),
+ 0L,
+ 2999L),
+ new StandardScalerModelData(
+ Vectors.dense(0.35, 1.1666667, 0),
+ Vectors.dense(1.5630099, 4.6654760, 1.5491933),
+ 1L,
+ 8999L),
+ new StandardScalerModelData(
+ Vectors.dense(0.3666667, 1.2222222, 0),
+ Vectors.dense(1.2369316, 3.7006005, 1.5),
+ 2L,
+ 11999L));
+
+ private static final double TOLERANCE = 1e-7;
+
+ private Table inputTable;
+
+ private Table inputTableWithProcessingTime;
+
+ private Table inputTableWithEventTime;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.getConfig().enableObjectReuse();
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ DataStream<Row> inputStream = env.fromCollection(inputData);
+ inputTable =
+ tEnv.fromDataStream(
+ inputStream,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .build())
+ .as("id", "input");
+
+ DataStream<Row> inputStreamWithProcessingTimeGap =
+ inputStream
+ .map(
+ new MapFunction<Row, Row>() {
+ private int count = 0;
+
+ @Override
+ public Row map(Row value) throws Exception {
+ count++;
+ if (count % 3 == 0) {
+ Thread.sleep(1000);
+ }
+ return value;
+ }
+ },
+ new RowTypeInfo(
+ new TypeInformation[] {
+ Types.LONG, DenseVectorTypeInfo.INSTANCE
+ },
+ new String[] {"id", "input"}))
+ .setParallelism(1);
+
+ inputTableWithProcessingTime = tEnv.fromDataStream(inputStreamWithProcessingTimeGap);
+
+ DataStream<Row> inputStreamWithEventTime =
+ inputStream.assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Row>forMonotonousTimestamps()
+ .withTimestampAssigner(
+ (SerializableTimestampAssigner<Row>)
+ (element, recordTimestamp) ->
+ element.getFieldAs(0)));
+ inputTableWithEventTime =
+ tEnv.fromDataStream(
+ inputStreamWithEventTime,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build())
+ .as("id", "input");
+ }
+
+ @Test
+ public void testParam() {
+ OnlineStandardScaler standardScaler = new OnlineStandardScaler();
+
+ assertEquals("input", standardScaler.getInputCol());
+ assertEquals(false, standardScaler.getWithMean());
+ assertEquals(true, standardScaler.getWithStd());
+ assertEquals("output", standardScaler.getOutputCol());
+ assertEquals("version", standardScaler.getModelVersionCol());
+ assertEquals(GlobalWindows.getInstance(), standardScaler.getWindows());
+ assertEquals(0L, standardScaler.getMaxAllowedModelDelayMs());
+
+ standardScaler
+ .setInputCol("test_input")
+ .setWithMean(true)
+ .setWithStd(false)
+ .setOutputCol("test_output")
+ .setModelVersionCol("model_version_col")
Review Comment:
Use non-default value here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1091380089
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/StandardScalerTest.java:
##########
@@ -206,7 +206,7 @@ public void testSaveLoadAndPredict() throws Exception {
model = TestUtils.saveAndReload(tEnv, model, tempFolder.newFolder().getAbsolutePath());
assertEquals(
- Arrays.asList("mean", "std"),
+ Arrays.asList("mean", "std", "version", "timestamp"),
Review Comment:
> What do you mean by verify model data contains those columns? Do you mean we only verify model data and leave model version and timestamp unchecked?
Yes that is what I means. IMO StandardScalerTest should verify the expected behavior of StandardScalerModel from the perspective of users who want to use StandardScalerModel. IMO users of StandardScalerModel should not need to know concepts such as model version or timestamp. Users should `OnlineStandardScaler` needs to know these information, and this behavior is covered by `OnlineStandardScalerTest`.
> I think letting StandardScalerModel have model version and timestamp is reasonable, since offline learning could be viewed as a special case of online learning
Yes we can tell users this. But is there benefit for users to take the extra effort to learn about this? If only the users of `OnlineStandardScaler` needs to know this, it seems more intuitive to put all related tests there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1066577618
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/OnlineStandardScalerTest.java:
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature;
+
+import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.common.window.CountTumblingWindows;
+import org.apache.flink.ml.common.window.EventTimeTumblingWindows;
+import org.apache.flink.ml.common.window.GlobalWindows;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScaler;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScalerModel;
+import org.apache.flink.ml.feature.standardscaler.StandardScalerModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.TestUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.test.util.AbstractTestBase;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests {@link OnlineStandardScaler} and {@link OnlineStandardScalerModel}. */
+public class OnlineStandardScalerTest extends AbstractTestBase {
+ @Rule public final TemporaryFolder tempFolder = new TemporaryFolder();
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private final List<Row> inputData =
+ Arrays.asList(
+ Row.of(0L, Vectors.dense(-2.5, 9, 1)),
+ Row.of(1000L, Vectors.dense(1.4, -5, 1)),
+ Row.of(2000L, Vectors.dense(2, -1, -2)),
+ Row.of(6000L, Vectors.dense(0.7, 3, 1)),
+ Row.of(7000L, Vectors.dense(0, 1, 1)),
+ Row.of(8000L, Vectors.dense(0.5, 0, -2)),
+ Row.of(9000L, Vectors.dense(0.4, 1, 1)),
+ Row.of(10000L, Vectors.dense(0.3, 2, 1)),
+ Row.of(11000L, Vectors.dense(0.5, 1, -2)));
+
+ private final List<StandardScalerModelData> expectedModelData =
+ Arrays.asList(
+ new StandardScalerModelData(
+ Vectors.dense(0.3, 1, 0),
+ Vectors.dense(2.4433583, 7.2111026, 1.7320508),
+ 0L,
+ 2999L),
+ new StandardScalerModelData(
+ Vectors.dense(0.35, 1.1666667, 0),
+ Vectors.dense(1.5630099, 4.6654760, 1.5491933),
+ 1L,
+ 8999L),
+ new StandardScalerModelData(
+ Vectors.dense(0.3666667, 1.2222222, 0),
+ Vectors.dense(1.2369316, 3.7006005, 1.5),
+ 2L,
+ 11999L));
+
+ private static final double TOLERANCE = 1e-7;
+
+ private Table inputTable;
+
+ private Table inputTableWithEventTime;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.getConfig().enableObjectReuse();
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ DataStream<Row> inputStream = env.fromCollection(inputData);
+ inputTable =
+ tEnv.fromDataStream(
+ inputStream,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .build())
+ .as("id", "input");
+
+ DataStream<Row> inputStreamWithEventTime =
+ inputStream.assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Row>forMonotonousTimestamps()
+ .withTimestampAssigner(
+ (SerializableTimestampAssigner<Row>)
+ (element, recordTimestamp) ->
+ element.getFieldAs(0)));
+ inputTableWithEventTime =
+ tEnv.fromDataStream(
+ inputStreamWithEventTime,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build())
+ .as("id", "input");
+ }
+
+ @Test
+ public void testParam() {
+ OnlineStandardScaler standardScaler = new OnlineStandardScaler();
+
+ assertEquals("input", standardScaler.getInputCol());
+ assertEquals(false, standardScaler.getWithMean());
+ assertEquals(true, standardScaler.getWithStd());
+ assertEquals("output", standardScaler.getOutputCol());
+ assertNull(standardScaler.getModelVersionCol());
+ assertEquals(GlobalWindows.getInstance(), standardScaler.getWindows());
+ assertEquals(0L, standardScaler.getMaxAllowedModelDelayMs());
+
+ standardScaler
+ .setInputCol("test_input")
+ .setWithMean(true)
+ .setWithStd(false)
+ .setOutputCol("test_output")
+ .setModelVersionCol("model_version_col")
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(3000)))
+ .setMaxAllowedModelDelayMs(3000L);
+
+ assertEquals("test_input", standardScaler.getInputCol());
+ assertEquals(true, standardScaler.getWithMean());
+ assertEquals(false, standardScaler.getWithStd());
+ assertEquals("test_output", standardScaler.getOutputCol());
+ assertEquals("model_version_col", standardScaler.getModelVersionCol());
+ assertEquals(
+ EventTimeTumblingWindows.of(Time.milliseconds(3000)), standardScaler.getWindows());
+ assertEquals(3000L, standardScaler.getMaxAllowedModelDelayMs());
+ }
+
+ @Test
+ public void testOutputSchema() {
+ Table renamedTable = inputTable.as("test_id", "test_input");
+ OnlineStandardScaler standardScaler =
+ new OnlineStandardScaler()
+ .setInputCol("test_input")
+ .setOutputCol("test_output")
+ .setModelVersionCol("model_version_col");
+ Table output = standardScaler.fit(renamedTable).transform(renamedTable)[0];
+
+ assertEquals(
+ Arrays.asList("test_id", "test_input", "test_output", "model_version_col"),
+ output.getResolvedSchema().getColumnNames());
+ }
+
+ @Test
+ public void testFitAndPredictWithEventTimeWindow() throws Exception {
+ OnlineStandardScaler standardScaler = new OnlineStandardScaler();
+ Table output;
+ int windowSizeMs = 3000;
+
+ // Tests event time window with maxAllowedModelDelayMs as 0.
+ standardScaler
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(windowSizeMs)))
+ .setModelVersionCol("modelVersionCol");
+ output = standardScaler.fit(inputTableWithEventTime).transform(inputTableWithEventTime)[0];
+ verifyUsedModelVersion(
Review Comment:
The prediction result is undetermined because we may use new model to predict old data.
So we can only check the model version here. The prediction logic is covered by other unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1093211142
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(
+ MLMetrics.ML_MODEL_GROUP,
+ OnlineStandardScalerModel.class.getSimpleName());
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() - maxAllowedModelDelayMs <= modelTimeStamp
Review Comment:
What is the value of `modelTimeStamp` if `processElement2()` has not been invoked? Should we explicitly give it a default value?
In general it is a good practice to make sure all variables are initialized in the constructor (or `open()` in this case) before they are used. Maybe do it for other variables as well.
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(
+ MLMetrics.ML_MODEL_GROUP,
+ OnlineStandardScalerModel.class.getSimpleName());
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() - maxAllowedModelDelayMs <= modelTimeStamp
Review Comment:
What is the value of `modelTimeStamp` if `processElement2()` has not been invoked? Should we explicitly give it a default value?
In general, it is a good practice to make sure all variables are initialized in the constructor (or `open()` in this case) before they are used. Maybe do it for other variables as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1090519401
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(MLMetrics.ML_MODEL_GROUP);
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() <= modelTimeStamp + maxAllowedModelDelayMs
+ && mean != null) {
+ doPrediction(dataPoint);
+ } else {
+ bufferedPointsState.add(dataPoint);
Review Comment:
There is a bit difference between `modelDataState` and `bufferedPointsState` --- There is one single element in model data in `modelDataState` and more than one element in `bufferedPointsState`.
Appending elements into liststate is an asyncronous operation and thus can help to improve the efficiency when caching many data points. On the other side, there is only one model data. If we update it using `state::clear(), state::add()`, it may introduce unnessary state operations (e.g., add it to state and remove it before a snapshot).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] jiangxin369 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by GitBox <gi...@apache.org>.
jiangxin369 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1068221091
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/OnlineStandardScalerTest.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature;
+
+import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.common.window.CountTumblingWindows;
+import org.apache.flink.ml.common.window.EventTimeTumblingWindows;
+import org.apache.flink.ml.common.window.GlobalWindows;
+import org.apache.flink.ml.common.window.ProcessingTimeTumblingWindows;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScaler;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScalerModel;
+import org.apache.flink.ml.feature.standardscaler.StandardScalerModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.TestUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.test.util.AbstractTestBase;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests {@link OnlineStandardScaler} and {@link OnlineStandardScalerModel}. */
+public class OnlineStandardScalerTest extends AbstractTestBase {
+ @Rule public final TemporaryFolder tempFolder = new TemporaryFolder();
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private final List<Row> inputData =
+ Arrays.asList(
+ Row.of(0L, Vectors.dense(-2.5, 9, 1)),
+ Row.of(1000L, Vectors.dense(1.4, -5, 1)),
+ Row.of(2000L, Vectors.dense(2, -1, -2)),
+ Row.of(6000L, Vectors.dense(0.7, 3, 1)),
+ Row.of(7000L, Vectors.dense(0, 1, 1)),
+ Row.of(8000L, Vectors.dense(0.5, 0, -2)),
+ Row.of(9000L, Vectors.dense(0.4, 1, 1)),
+ Row.of(10000L, Vectors.dense(0.3, 2, 1)),
+ Row.of(11000L, Vectors.dense(0.5, 1, -2)));
+
+ private final List<StandardScalerModelData> expectedModelData =
+ Arrays.asList(
+ new StandardScalerModelData(
+ Vectors.dense(0.3, 1, 0),
+ Vectors.dense(2.4433583, 7.2111026, 1.7320508),
+ 0L,
+ 2999L),
+ new StandardScalerModelData(
+ Vectors.dense(0.35, 1.1666667, 0),
+ Vectors.dense(1.5630099, 4.6654760, 1.5491933),
+ 1L,
+ 8999L),
+ new StandardScalerModelData(
+ Vectors.dense(0.3666667, 1.2222222, 0),
+ Vectors.dense(1.2369316, 3.7006005, 1.5),
+ 2L,
+ 11999L));
+
+ private static final double TOLERANCE = 1e-7;
+
+ private Table inputTable;
+
+ private Table inputTableWithProcessingTime;
+
+ private Table inputTableWithEventTime;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.getConfig().enableObjectReuse();
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ DataStream<Row> inputStream = env.fromCollection(inputData);
+ inputTable =
+ tEnv.fromDataStream(
+ inputStream,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .build())
+ .as("id", "input");
+
+ DataStream<Row> inputStreamWithProcessingTimeGap =
+ inputStream
+ .map(
+ new MapFunction<Row, Row>() {
+ private int count = 0;
+
+ @Override
+ public Row map(Row value) throws Exception {
+ count++;
+ if (count % 3 == 0) {
+ Thread.sleep(1000);
+ }
+ return value;
+ }
+ },
+ new RowTypeInfo(
+ new TypeInformation[] {
+ Types.LONG, DenseVectorTypeInfo.INSTANCE
+ },
+ new String[] {"id", "input"}))
+ .setParallelism(1);
+
+ inputTableWithProcessingTime = tEnv.fromDataStream(inputStreamWithProcessingTimeGap);
+
+ DataStream<Row> inputStreamWithEventTime =
+ inputStream.assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Row>forMonotonousTimestamps()
+ .withTimestampAssigner(
+ (SerializableTimestampAssigner<Row>)
+ (element, recordTimestamp) ->
+ element.getFieldAs(0)));
+ inputTableWithEventTime =
+ tEnv.fromDataStream(
+ inputStreamWithEventTime,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build())
+ .as("id", "input");
+ }
+
+ @Test
+ public void testParam() {
+ OnlineStandardScaler standardScaler = new OnlineStandardScaler();
+
+ assertEquals("input", standardScaler.getInputCol());
+ assertEquals(false, standardScaler.getWithMean());
+ assertEquals(true, standardScaler.getWithStd());
+ assertEquals("output", standardScaler.getOutputCol());
+ assertNull(standardScaler.getModelVersionCol());
+ assertEquals(GlobalWindows.getInstance(), standardScaler.getWindows());
+ assertEquals(0L, standardScaler.getMaxAllowedModelDelayMs());
+
+ standardScaler
+ .setInputCol("test_input")
+ .setWithMean(true)
+ .setWithStd(false)
+ .setOutputCol("test_output")
+ .setModelVersionCol("model_version_col")
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(3000)))
+ .setMaxAllowedModelDelayMs(3000L);
+
+ assertEquals("test_input", standardScaler.getInputCol());
+ assertEquals(true, standardScaler.getWithMean());
+ assertEquals(false, standardScaler.getWithStd());
+ assertEquals("test_output", standardScaler.getOutputCol());
+ assertEquals("model_version_col", standardScaler.getModelVersionCol());
+ assertEquals(
+ EventTimeTumblingWindows.of(Time.milliseconds(3000)), standardScaler.getWindows());
+ assertEquals(3000L, standardScaler.getMaxAllowedModelDelayMs());
+ }
+
+ @Test
+ public void testOutputSchema() {
+ Table renamedTable = inputTable.as("test_id", "test_input");
+ OnlineStandardScaler standardScaler =
+ new OnlineStandardScaler()
+ .setInputCol("test_input")
+ .setOutputCol("test_output")
+ .setModelVersionCol("model_version_col");
Review Comment:
Could you also test the output schema without `setModelVersionCol`? So as in pytest.
##########
docs/content/docs/operators/feature/onlinestandardscaler.md:
##########
@@ -0,0 +1,261 @@
+---
+title: "OnlineStandardScaler"
+weight: 1
+type: docs
+aliases:
+- /operators/feature/onlinestandardscaler.html
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## OnlineStandardScaler
+
+An Estimator which implements the online standard scaling algorithm, which
+is the online version of StandardScaler.
+
+OnlineStandardScaler splits the input data by the user-specified window strategy.
+For each window, it computes the mean and standard deviation using the data seen
+so far (i.e., not only the data in the current window, but also the history data).
+The model data generated by OnlineStandardScaler is a model stream.
+There is one model data for each window.
+
+During the inference phase (i.e., using OnlineStandardScalerModel for prediction),
+users could output the model version that is used for predicting each data point.
+Moreover,
+- When the train data and test data both contain event time, users could
+specify the maximum difference between timestamp of the input and model data,
Review Comment:
```suggestion
specify the maximum difference between the timestamps of the input and model data,
```
There are still several same comments not addressed
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/OnlineStandardScalerTest.java:
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature;
+
+import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.common.window.CountTumblingWindows;
+import org.apache.flink.ml.common.window.EventTimeTumblingWindows;
+import org.apache.flink.ml.common.window.GlobalWindows;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScaler;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScalerModel;
+import org.apache.flink.ml.feature.standardscaler.StandardScalerModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.TestUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.test.util.AbstractTestBase;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests {@link OnlineStandardScaler} and {@link OnlineStandardScalerModel}. */
+public class OnlineStandardScalerTest extends AbstractTestBase {
+ @Rule public final TemporaryFolder tempFolder = new TemporaryFolder();
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private final List<Row> inputData =
+ Arrays.asList(
+ Row.of(0L, Vectors.dense(-2.5, 9, 1)),
+ Row.of(1000L, Vectors.dense(1.4, -5, 1)),
+ Row.of(2000L, Vectors.dense(2, -1, -2)),
+ Row.of(6000L, Vectors.dense(0.7, 3, 1)),
+ Row.of(7000L, Vectors.dense(0, 1, 1)),
+ Row.of(8000L, Vectors.dense(0.5, 0, -2)),
+ Row.of(9000L, Vectors.dense(0.4, 1, 1)),
+ Row.of(10000L, Vectors.dense(0.3, 2, 1)),
+ Row.of(11000L, Vectors.dense(0.5, 1, -2)));
+
+ private final List<StandardScalerModelData> expectedModelData =
+ Arrays.asList(
+ new StandardScalerModelData(
+ Vectors.dense(0.3, 1, 0),
+ Vectors.dense(2.4433583, 7.2111026, 1.7320508),
+ 0L,
+ 2999L),
+ new StandardScalerModelData(
+ Vectors.dense(0.35, 1.1666667, 0),
+ Vectors.dense(1.5630099, 4.6654760, 1.5491933),
+ 1L,
+ 8999L),
+ new StandardScalerModelData(
+ Vectors.dense(0.3666667, 1.2222222, 0),
+ Vectors.dense(1.2369316, 3.7006005, 1.5),
+ 2L,
+ 11999L));
+
+ private static final double TOLERANCE = 1e-7;
+
+ private Table inputTable;
+
+ private Table inputTableWithEventTime;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.getConfig().enableObjectReuse();
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ DataStream<Row> inputStream = env.fromCollection(inputData);
+ inputTable =
+ tEnv.fromDataStream(
+ inputStream,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .build())
+ .as("id", "input");
+
+ DataStream<Row> inputStreamWithEventTime =
+ inputStream.assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Row>forMonotonousTimestamps()
+ .withTimestampAssigner(
+ (SerializableTimestampAssigner<Row>)
+ (element, recordTimestamp) ->
+ element.getFieldAs(0)));
+ inputTableWithEventTime =
+ tEnv.fromDataStream(
+ inputStreamWithEventTime,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build())
+ .as("id", "input");
+ }
+
+ @Test
+ public void testParam() {
+ OnlineStandardScaler standardScaler = new OnlineStandardScaler();
+
+ assertEquals("input", standardScaler.getInputCol());
+ assertEquals(false, standardScaler.getWithMean());
+ assertEquals(true, standardScaler.getWithStd());
+ assertEquals("output", standardScaler.getOutputCol());
+ assertNull(standardScaler.getModelVersionCol());
+ assertEquals(GlobalWindows.getInstance(), standardScaler.getWindows());
+ assertEquals(0L, standardScaler.getMaxAllowedModelDelayMs());
+
+ standardScaler
+ .setInputCol("test_input")
+ .setWithMean(true)
+ .setWithStd(false)
+ .setOutputCol("test_output")
+ .setModelVersionCol("model_version_col")
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(3000)))
+ .setMaxAllowedModelDelayMs(3000L);
+
+ assertEquals("test_input", standardScaler.getInputCol());
+ assertEquals(true, standardScaler.getWithMean());
+ assertEquals(false, standardScaler.getWithStd());
+ assertEquals("test_output", standardScaler.getOutputCol());
+ assertEquals("model_version_col", standardScaler.getModelVersionCol());
+ assertEquals(
+ EventTimeTumblingWindows.of(Time.milliseconds(3000)), standardScaler.getWindows());
+ assertEquals(3000L, standardScaler.getMaxAllowedModelDelayMs());
+ }
+
+ @Test
+ public void testOutputSchema() {
+ Table renamedTable = inputTable.as("test_id", "test_input");
+ OnlineStandardScaler standardScaler =
+ new OnlineStandardScaler()
+ .setInputCol("test_input")
+ .setOutputCol("test_output")
+ .setModelVersionCol("model_version_col");
+ Table output = standardScaler.fit(renamedTable).transform(renamedTable)[0];
+
+ assertEquals(
+ Arrays.asList("test_id", "test_input", "test_output", "model_version_col"),
+ output.getResolvedSchema().getColumnNames());
+ }
+
+ @Test
+ public void testFitAndPredictWithEventTimeWindow() throws Exception {
+ OnlineStandardScaler standardScaler = new OnlineStandardScaler();
+ Table output;
+ int windowSizeMs = 3000;
+
+ // Tests event time window with maxAllowedModelDelayMs as 0.
+ standardScaler
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(windowSizeMs)))
+ .setModelVersionCol("modelVersionCol");
+ output = standardScaler.fit(inputTableWithEventTime).transform(inputTableWithEventTime)[0];
+ verifyUsedModelVersion(
Review Comment:
I understand, thanks for the explanation.
##########
docs/content/docs/operators/feature/onlinestandardscaler.md:
##########
@@ -0,0 +1,261 @@
+---
+title: "OnlineStandardScaler"
+weight: 1
+type: docs
+aliases:
+- /operators/feature/onlinestandardscaler.html
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## OnlineStandardScaler
+
+An Estimator which implements the online standard scaling algorithm, which
+is the online version of StandardScaler.
+
+OnlineStandardScaler splits the input data by the user-specified window strategy.
+For each window, it computes the mean and standard deviation using the data seen
+so far (i.e., not only the data in the current window, but also the history data).
+The model data generated by OnlineStandardScaler is a model stream.
+There is one model data for each window.
+
+During the inference phase (i.e., using OnlineStandardScalerModel for prediction),
+users could output the model version that is used for predicting each data point.
+Moreover,
+- When the train data and test data both contains event time, users could
+specify the maximum difference between timestamp of the input and model data,
+which enforces to use a relatively fresh model for prediction.
+- Otherwise, the prediction process always use the current model data for prediction.
+
+
+### Input Columns
+
+| Param name | Type | Default | Description |
+|:-----------|:-------|:----------|:-----------------------|
+| inputCol | Vector | `"input"` | Features to be scaled. |
+
+### Output Columns
+
+| Param name | Type | Default | Description |
+|:-----------|:-------|:-----------|:-----------------|
+| outputCol | Vector | `"output"` | Scaled features. |
+
+### Parameters
+
+Below are the parameters required by `OnlineStandardScalerModel`.
+
+| Key | Default | Type | Required | Description |
+|------------------------|------------|---------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| inputCol | `"input"` | String | no | Input column name. |
+| outputCol | `"output"` | String | no | Output column name. |
+| withMean | `false` | Boolean | no | Whether centers the data with mean before scaling. |
+| withStd | `true` | Boolean | no | Whether scales the data with standard deviation. |
+ | modelVersionCol | `null` | String | no | The version of the model data that the input data is predicted with. |
+ | maxAllowedModelDelayMs | `0L` | Long | no | The maximum difference between timestamp of the input record and model data when using the model data to predict that input record. This param only works when the input contains event time." |
+
+`OnlineStandardScaler` needs parameters above and also below.
+
+| Key | Default | Type | Required | Description |
+|---------|-------------------------------|---------|----------|--------------------------------------------------------------------------------|
+| windows | `GlobalWindows.getInstance()` | Windows | no | Windowing strategy that determines how to create mini-batches from input data. |
+
+
+### Examples
+
+{{< tabs examples >}}
+
+{{< tab "Java">}}
+
+```java
+import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.ml.common.window.EventTimeTumblingWindows;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScaler;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScalerModel;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import java.util.Arrays;
+import java.util.List;
+
+/** Simple program that trains a OnlineStandardScaler model and uses it for feature engineering. */
+public class OnlineStandardScalerExample {
+ public static void main(String[] args) {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ // Generates input data.
+ List<Row> inputData =
+ Arrays.asList(
+ Row.of(0L, Vectors.dense(-2.5, 9, 1)),
+ Row.of(1000L, Vectors.dense(1.4, -5, 1)),
+ Row.of(2000L, Vectors.dense(2, -1, -2)),
+ Row.of(6000L, Vectors.dense(0.7, 3, 1)),
+ Row.of(7000L, Vectors.dense(0, 1, 1)),
+ Row.of(8000L, Vectors.dense(0.5, 0, -2)),
+ Row.of(9000L, Vectors.dense(0.4, 1, 1)),
+ Row.of(10000L, Vectors.dense(0.3, 2, 1)),
+ Row.of(11000L, Vectors.dense(0.5, 1, -2)));
+
+ DataStream<Row> inputStream = env.fromCollection(inputData);
+
+ DataStream<Row> inputStreamWithEventTime =
+ inputStream.assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Row>forMonotonousTimestamps()
+ .withTimestampAssigner(
+ (SerializableTimestampAssigner<Row>)
+ (element, recordTimestamp) ->
+ element.getFieldAs(0)));
+
+ Table inputTable =
+ tEnv.fromDataStream(
+ inputStreamWithEventTime,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build())
+ .as("id", "input");
+
+ // Creates an OnlineStandardScaler object and initializes its parameters.
+ long windowSizeMs = 3000;
+ OnlineStandardScaler onlineStandardScaler =
+ new OnlineStandardScaler()
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(windowSizeMs)))
+ .setModelVersionCol("modelVersionCol");
+
+ // Trains the OnlineStandardScaler Model.
+ OnlineStandardScalerModel model = onlineStandardScaler.fit(inputTable);
+
+ // Uses the OnlineStandardScaler Model for predictions.
+ Table outputTable = model.transform(inputTable)[0];
+
+ // Extracts and displays the results.
+ for (CloseableIterator<Row> it = outputTable.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ DenseVector inputValue = (DenseVector) row.getField(onlineStandardScaler.getInputCol());
+ DenseVector outputValue =
+ (DenseVector) row.getField(onlineStandardScaler.getOutputCol());
+ long modelVersion = row.getFieldAs(onlineStandardScaler.getModelVersionCol());
+ System.out.printf(
+ "Input Value: %s\tOutput Value: %s\tModel Version: %s\n",
+ inputValue, outputValue, modelVersion);
+ }
+ }
+}
+
Review Comment:
Is this addressed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1091394858
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasModelVersionCol.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.StringParam;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared model version column param. */
+public interface HasModelVersionCol<T> extends WithParams<T> {
+ Param<String> MODEL_VERSION_COL =
+ new StringParam(
+ "modelVersionCol",
+ "The version of the model data that the input data is predicted with.",
+ null);
Review Comment:
By `the exiting prediction case`, do you mean `HasPredictionCol`? Currently `HasPredictionCol` defaults to prediction. Can you clarify it here?
The benefit is to minimize the config that users need to tune for the common scenario targeted by algorithms that depends on `HasModelVersionCol`.
I suppose that for algorithms such as `OnlineStandardScaler`, their users will run the algorithm continuously and observe multiple models emitted by it. And it seems useful for users to know "which version of the model data that the input data is predicted with". Since there is observable benefit and negligible cost, it seems better to enable this column by default?
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasModelVersionCol.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.StringParam;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared model version column param. */
+public interface HasModelVersionCol<T> extends WithParams<T> {
+ Param<String> MODEL_VERSION_COL =
+ new StringParam(
+ "modelVersionCol",
+ "The version of the model data that the input data is predicted with.",
+ null);
Review Comment:
By `the exiting prediction case`, do you mean `HasPredictionCol`? Currently `HasPredictionCol` defaults to prediction. Can you clarify it here?
The benefit is to minimize the config that users need to tune for the common scenario targeted by algorithms that depends on `HasModelVersionCol`.
I suppose that for algorithms such as `OnlineStandardScaler`, their users will run the algorithm continuously and observe multiple models emitted by it. And it seems useful for users to know "which version of the model data that the input data is predicted with". Since there is observable usability benefit and negligible cost, it seems better to enable this column by default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1090525929
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/StandardScalerTest.java:
##########
@@ -206,7 +206,7 @@ public void testSaveLoadAndPredict() throws Exception {
model = TestUtils.saveAndReload(tEnv, model, tempFolder.newFolder().getAbsolutePath());
assertEquals(
- Arrays.asList("mean", "std"),
+ Arrays.asList("mean", "std", "version", "timestamp"),
Review Comment:
Well, I think letting `StandardScalerModel` have model version and timestamp is reasonable, since offline learning could be viewed as a special case of online learning. What we need to do is to make sure that version and timestamp has a clear meaning. For example, the version of offline model is always zero and time stamp is `LONG.MAX_VALUE`.
What do you mean by verify `model data contains those columns`? Do you mean we only verify `model data` and leave `model version` and `timestamp` unchecked?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1094445447
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(
+ MLMetrics.ML_MODEL_GROUP,
+ OnlineStandardScalerModel.class.getSimpleName());
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() - maxAllowedModelDelayMs <= modelTimeStamp
Review Comment:
The default value should be zero here. However, it is hard to tell what the `modelTimeStamp` should be if there is no model data. So I leave it as un-initialized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by GitBox <gi...@apache.org>.
lindong28 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1070769249
##########
flink-ml-core/src/main/java/org/apache/flink/ml/common/metrics/MLMetrics.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.metrics;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+/**
+ * A collection class for handling metrics in Flink ML.
+ *
+ * <p>All metrics of Flink ML are registered under group "ml", which is a child group of {@link
+ * org.apache.flink.metrics.groups.OperatorMetricGroup}. Metrics related to model data will be
+ * registered in the group "ml.model".
+ *
+ * <p>For example, the timestamp of the current model data will be reported in metric:
+ * "{some_parent_groups}.operator.ml.model.timestamp". And the version of the current model data
+ * will be reported in metric: "{some_parent_groups}.operator.ml.model.version".
+ */
+@PublicEvolving
Review Comment:
Since we have not opened FLIP for these metrics, should we mark it `@Experimental` for now?
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasModelVersionCol.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.StringParam;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared model version column param. */
+public interface HasModelVersionCol<T> extends WithParams<T> {
+ Param<String> MODEL_VERSION_COL =
+ new StringParam(
+ "modelVersionCol",
+ "The version of the model data that the input data is predicted with.",
Review Comment:
Would it be useful to also specify the value type (e.g. long) of this column?
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasMaxAllowedModelDelayMs.java:
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.LongParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared max allowed model delay in milliseconds param. */
+public interface HasMaxAllowedModelDelayMs<T> extends WithParams<T> {
+ Param<Long> MAX_ALLOWED_MODEL_DELAY_MS =
+ new LongParam(
+ "maxAllowedModelDelayMs",
+ "The maximum difference between the timestamps of the input record and model data when "
Review Comment:
It seems more readable to make the following change:
`model data when using the model data to predict that input record`
->
`the model data that is used to make predictions on that input record`
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/StandardScalerTest.java:
##########
@@ -206,7 +206,7 @@ public void testSaveLoadAndPredict() throws Exception {
model = TestUtils.saveAndReload(tEnv, model, tempFolder.newFolder().getAbsolutePath());
assertEquals(
- Arrays.asList("mean", "std"),
+ Arrays.asList("mean", "std", "version", "timestamp"),
Review Comment:
Even though we updated the model data emitted by StandardScalerModel that have these columns, it is an unintended implementation detail and is avoided, since we didn't expect `StandardScalerModel` to have concepts of version.
It might be more intuitive to verify `model data contains those columns`. What do you think?
Same for other changes in this test class that can not be explained by the semantics of the StandardScaler algorithm.
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(MLMetrics.ML_MODEL_GROUP);
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() <= modelTimeStamp + maxAllowedModelDelayMs
+ && mean != null) {
+ doPrediction(dataPoint);
+ } else {
+ bufferedPointsState.add(dataPoint);
+ }
+ }
+
+ @Override
+ public void processElement2(StreamRecord<StandardScalerModelData> streamRecord)
+ throws Exception {
+ modelData = streamRecord.getValue();
+
+ modelTimeStamp = modelData.timestamp;
+ modelVersion = modelData.version;
+ mean = modelData.mean;
+ DenseVector std = modelData.std;
+
+ if (withStd) {
+ scale = std;
+ double[] scaleValues = scale.values;
+ for (int i = 0; i < scaleValues.length; i++) {
+ scaleValues[i] = scaleValues[i] == 0 ? 0 : 1 / scaleValues[i];
+ }
+ }
+
+ // Does prediction on the cached data.
+ List<StreamRecord> unprocessedElements = new ArrayList<>();
+ for (StreamRecord dataPoint : bufferedPointsState.get()) {
+ if (dataPoint.getTimestamp() <= modelTimeStamp + maxAllowedModelDelayMs) {
+ doPrediction(dataPoint);
+ } else {
+ unprocessedElements.add(dataPoint);
+ }
+ }
+ bufferedPointsState.clear();
+ if (unprocessedElements.size() > 0) {
+ bufferedPointsState.update(unprocessedElements);
Review Comment:
As a minor optimization, can we only clear/update the state if at least one element is predicted in the above for loop?
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasMaxAllowedModelDelayMs.java:
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.LongParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared max allowed model delay in milliseconds param. */
+public interface HasMaxAllowedModelDelayMs<T> extends WithParams<T> {
+ Param<Long> MAX_ALLOWED_MODEL_DELAY_MS =
+ new LongParam(
+ "maxAllowedModelDelayMs",
+ "The maximum difference between the timestamps of the input record and model data when "
Review Comment:
`maximum difference` -> `maximum difference allowed`
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(MLMetrics.ML_MODEL_GROUP);
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() <= modelTimeStamp + maxAllowedModelDelayMs
+ && mean != null) {
+ doPrediction(dataPoint);
+ } else {
+ bufferedPointsState.add(dataPoint);
Review Comment:
Given that we only update `modelDataState` in `snapshotState()`, I suppose it is because updating `modelDataState()` every time we receive a model data is more expensive than updating the in-memory `modelData`.
Should we follow the same approach for `dataPoint` consistency and performance?
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasMaxAllowedModelDelayMs.java:
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.LongParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared max allowed model delay in milliseconds param. */
+public interface HasMaxAllowedModelDelayMs<T> extends WithParams<T> {
+ Param<Long> MAX_ALLOWED_MODEL_DELAY_MS =
+ new LongParam(
+ "maxAllowedModelDelayMs",
+ "The maximum difference between the timestamps of the input record and model data when "
Review Comment:
`maximum difference` -> `maximum difference allowed`
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(MLMetrics.ML_MODEL_GROUP);
Review Comment:
Since we might have multiple models running in the same Flink Job, it might be better to specify the model name to reduce chance of conflict.
How about using `addGroup(MLMetrics.ML_MODEL_GROUP, "OnlineStandardScalerModel")` here?
##########
docs/content/docs/operators/feature/onlinestandardscaler.md:
##########
@@ -0,0 +1,260 @@
+---
+title: "OnlineStandardScaler"
+weight: 1
+type: docs
+aliases:
+- /operators/feature/onlinestandardscaler.html
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## OnlineStandardScaler
+
+An Estimator which implements the online standard scaling algorithm, which
+is the online version of StandardScaler.
+
+OnlineStandardScaler splits the input data by the user-specified window strategy.
+For each window, it computes the mean and standard deviation using the data seen
+so far (i.e., not only the data in the current window, but also the history data).
+The model data generated by OnlineStandardScaler is a model stream.
+There is one model data for each window.
+
+During the inference phase (i.e., using OnlineStandardScalerModel for prediction),
+users could output the model version that is used for predicting each data point.
+Moreover,
+- When the train data and test data both contain event time, users could
+specify the maximum difference between the timestamps of the input and model data,
+which enforces to use a relatively fresh model for prediction.
+- Otherwise, the prediction process always uses the current model data for prediction.
+
+
+### Input Columns
+
+| Param name | Type | Default | Description |
+|:-----------|:-------|:----------|:-----------------------|
+| inputCol | Vector | `"input"` | Features to be scaled. |
+
+### Output Columns
+
+| Param name | Type | Default | Description |
+|:-----------|:-------|:-----------|:-----------------|
+| outputCol | Vector | `"output"` | Scaled features. |
Review Comment:
Should we add `modelVersionCol` here?
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(MLMetrics.ML_MODEL_GROUP);
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() <= modelTimeStamp + maxAllowedModelDelayMs
Review Comment:
According to the maxAllowedModelDelayMs doc, this parameter should only be used if the input contains event time. I suppose there should be logic that differentiates between event time and system time. Can you explain where is this logic?
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasModelVersionCol.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.StringParam;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared model version column param. */
+public interface HasModelVersionCol<T> extends WithParams<T> {
+ Param<String> MODEL_VERSION_COL =
+ new StringParam(
+ "modelVersionCol",
+ "The version of the model data that the input data is predicted with.",
+ null);
Review Comment:
Would it be more usable to using "version" as the default value and always output this information? The reason is that I suppose adding an extra long column should not have an observable performance impact for nearline (not batch) prediction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1066575361
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScaler.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.ProcessAllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * An Estimator which implements the online standard scaling algorithm, which is the online version
+ * of {@link StandardScaler}.
+ *
+ * <p>OnlineStandardScaler splits the input data by the user-specified window strategy (i.e., {@link
+ * org.apache.flink.ml.common.param.HasWindows}). For each window, it computes the mean and standard
+ * deviation using the data seen so far (i.e., not only the data in the current window, but also the
+ * history data). The model data generated by OnlineStandardScaler is a model stream. There is one
+ * model data for each window.
+ *
+ * <p>During the inference phase (i.e., using {@link OnlineStandardScalerModel} for prediction),
+ * users could output the model version that is used for predicting each data point. Moreover,
+ *
+ * <ul>
+ * <li>When the train data and test data both contains event time, users could specify the maximum
+ * difference between timestamp of the input and model data ({@link
+ * org.apache.flink.ml.common.param.HasMaxAllowedModelDelayMs}), which enforces to use a
+ * relatively fresh model for prediction.
+ * <li>Otherwise, the prediction process always use the current model data for prediction.
+ * </ul>
+ */
+public class OnlineStandardScaler
+ implements Estimator<OnlineStandardScaler, OnlineStandardScalerModel>,
+ OnlineStandardScalerParams<OnlineStandardScaler> {
+
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public OnlineStandardScaler() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public OnlineStandardScalerModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<StandardScalerModelData> modelData =
+ DataStreamUtils.windowAllAndProcess(
+ tEnv.toDataStream(inputs[0]),
+ getWindows(),
+ new ComputeModelDataFunction<>(getInputCol()));
+
+ OnlineStandardScalerModel model =
+ new OnlineStandardScalerModel().setModelData(tEnv.fromDataStream(modelData));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ private static class ComputeModelDataFunction<W extends Window>
+ extends ProcessAllWindowFunction<Row, StandardScalerModelData, W> {
+
+ private final String inputCol;
+
+ public ComputeModelDataFunction(String inputCol) {
+ this.inputCol = inputCol;
+ }
+
+ @Override
+ public void process(
+ ProcessAllWindowFunction<Row, StandardScalerModelData, W>.Context context,
+ Iterable<Row> iterable,
+ Collector<StandardScalerModelData> collector)
+ throws Exception {
+ ListState<DenseVector> sumState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>(
+ "sumState", DenseVectorTypeInfo.INSTANCE));
+ ListState<DenseVector> squaredSumState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>(
+ "squaredSumState", DenseVectorTypeInfo.INSTANCE));
+ ListState<Long> numElementsState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>("numElementsState", Types.LONG));
+ ListState<Long> modelVersionState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>("modelVersionState", Types.LONG));
+ DenseVector sum =
+ OperatorStateUtils.getUniqueElement(sumState, "sumState").orElse(null);
+ DenseVector squaredSum =
+ OperatorStateUtils.getUniqueElement(squaredSumState, "squaredSumState")
+ .orElse(null);
+ long numElements =
+ OperatorStateUtils.getUniqueElement(numElementsState, "numElementsState")
+ .orElse(0L);
+ long modelVersion =
+ OperatorStateUtils.getUniqueElement(modelVersionState, "modelVersionState")
+ .orElse(0L);
+
+ long numElementsBefore = numElements;
+ for (Row element : iterable) {
+ Vector inputVec =
+ ((Vector) Objects.requireNonNull(element.getField(inputCol))).clone();
Review Comment:
Because we need to do some in-place operations in Line#154, i.e., `BLAS.hDot(inputVec, inputVec);`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1092935488
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/StandardScalerTest.java:
##########
@@ -206,7 +206,7 @@ public void testSaveLoadAndPredict() throws Exception {
model = TestUtils.saveAndReload(tEnv, model, tempFolder.newFolder().getAbsolutePath());
assertEquals(
- Arrays.asList("mean", "std"),
+ Arrays.asList("mean", "std", "version", "timestamp"),
Review Comment:
Thanks for the explanation. I have updated the PR as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1090535539
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasModelVersionCol.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.StringParam;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared model version column param. */
+public interface HasModelVersionCol<T> extends WithParams<T> {
+ Param<String> MODEL_VERSION_COL =
+ new StringParam(
+ "modelVersionCol",
+ "The version of the model data that the input data is predicted with.",
+ null);
Review Comment:
I agree that it should not raise an observable performance impact here. I set the default value as `null` since it is consistent with the exiting prediction case. Is there any other benefit (except debugging) to output the model version by default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] jiangxin369 commented on pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by GitBox <gi...@apache.org>.
jiangxin369 commented on PR #196:
URL: https://github.com/apache/flink-ml/pull/196#issuecomment-1383146696
@zhipeng93 Thanks for the updates, LGTM. @lindong28 Could you take a look at the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1090392883
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(MLMetrics.ML_MODEL_GROUP);
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() <= modelTimeStamp + maxAllowedModelDelayMs
Review Comment:
Good catch, it is indeed a bug. I have fixed the training logic as follows:
- When the window strategy is based on event-time, the timestamp of model data is the maximum timestamp of the window that generates this model.
- Otherwise, the timestamp of the model data is `LONG.MAX_VALUE`.
Moreover, to avoid the possible overflow, the `if condition` here is updated as `if (dataPoint.getTimestamp() - maxAllowedModelDelayMs <= modelTimeStamp`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1066577618
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/OnlineStandardScalerTest.java:
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature;
+
+import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.common.window.CountTumblingWindows;
+import org.apache.flink.ml.common.window.EventTimeTumblingWindows;
+import org.apache.flink.ml.common.window.GlobalWindows;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScaler;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScalerModel;
+import org.apache.flink.ml.feature.standardscaler.StandardScalerModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.TestUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.test.util.AbstractTestBase;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests {@link OnlineStandardScaler} and {@link OnlineStandardScalerModel}. */
+public class OnlineStandardScalerTest extends AbstractTestBase {
+ @Rule public final TemporaryFolder tempFolder = new TemporaryFolder();
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private final List<Row> inputData =
+ Arrays.asList(
+ Row.of(0L, Vectors.dense(-2.5, 9, 1)),
+ Row.of(1000L, Vectors.dense(1.4, -5, 1)),
+ Row.of(2000L, Vectors.dense(2, -1, -2)),
+ Row.of(6000L, Vectors.dense(0.7, 3, 1)),
+ Row.of(7000L, Vectors.dense(0, 1, 1)),
+ Row.of(8000L, Vectors.dense(0.5, 0, -2)),
+ Row.of(9000L, Vectors.dense(0.4, 1, 1)),
+ Row.of(10000L, Vectors.dense(0.3, 2, 1)),
+ Row.of(11000L, Vectors.dense(0.5, 1, -2)));
+
+ private final List<StandardScalerModelData> expectedModelData =
+ Arrays.asList(
+ new StandardScalerModelData(
+ Vectors.dense(0.3, 1, 0),
+ Vectors.dense(2.4433583, 7.2111026, 1.7320508),
+ 0L,
+ 2999L),
+ new StandardScalerModelData(
+ Vectors.dense(0.35, 1.1666667, 0),
+ Vectors.dense(1.5630099, 4.6654760, 1.5491933),
+ 1L,
+ 8999L),
+ new StandardScalerModelData(
+ Vectors.dense(0.3666667, 1.2222222, 0),
+ Vectors.dense(1.2369316, 3.7006005, 1.5),
+ 2L,
+ 11999L));
+
+ private static final double TOLERANCE = 1e-7;
+
+ private Table inputTable;
+
+ private Table inputTableWithEventTime;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.getConfig().enableObjectReuse();
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ DataStream<Row> inputStream = env.fromCollection(inputData);
+ inputTable =
+ tEnv.fromDataStream(
+ inputStream,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .build())
+ .as("id", "input");
+
+ DataStream<Row> inputStreamWithEventTime =
+ inputStream.assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Row>forMonotonousTimestamps()
+ .withTimestampAssigner(
+ (SerializableTimestampAssigner<Row>)
+ (element, recordTimestamp) ->
+ element.getFieldAs(0)));
+ inputTableWithEventTime =
+ tEnv.fromDataStream(
+ inputStreamWithEventTime,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build())
+ .as("id", "input");
+ }
+
+ @Test
+ public void testParam() {
+ OnlineStandardScaler standardScaler = new OnlineStandardScaler();
+
+ assertEquals("input", standardScaler.getInputCol());
+ assertEquals(false, standardScaler.getWithMean());
+ assertEquals(true, standardScaler.getWithStd());
+ assertEquals("output", standardScaler.getOutputCol());
+ assertNull(standardScaler.getModelVersionCol());
+ assertEquals(GlobalWindows.getInstance(), standardScaler.getWindows());
+ assertEquals(0L, standardScaler.getMaxAllowedModelDelayMs());
+
+ standardScaler
+ .setInputCol("test_input")
+ .setWithMean(true)
+ .setWithStd(false)
+ .setOutputCol("test_output")
+ .setModelVersionCol("model_version_col")
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(3000)))
+ .setMaxAllowedModelDelayMs(3000L);
+
+ assertEquals("test_input", standardScaler.getInputCol());
+ assertEquals(true, standardScaler.getWithMean());
+ assertEquals(false, standardScaler.getWithStd());
+ assertEquals("test_output", standardScaler.getOutputCol());
+ assertEquals("model_version_col", standardScaler.getModelVersionCol());
+ assertEquals(
+ EventTimeTumblingWindows.of(Time.milliseconds(3000)), standardScaler.getWindows());
+ assertEquals(3000L, standardScaler.getMaxAllowedModelDelayMs());
+ }
+
+ @Test
+ public void testOutputSchema() {
+ Table renamedTable = inputTable.as("test_id", "test_input");
+ OnlineStandardScaler standardScaler =
+ new OnlineStandardScaler()
+ .setInputCol("test_input")
+ .setOutputCol("test_output")
+ .setModelVersionCol("model_version_col");
+ Table output = standardScaler.fit(renamedTable).transform(renamedTable)[0];
+
+ assertEquals(
+ Arrays.asList("test_id", "test_input", "test_output", "model_version_col"),
+ output.getResolvedSchema().getColumnNames());
+ }
+
+ @Test
+ public void testFitAndPredictWithEventTimeWindow() throws Exception {
+ OnlineStandardScaler standardScaler = new OnlineStandardScaler();
+ Table output;
+ int windowSizeMs = 3000;
+
+ // Tests event time window with maxAllowedModelDelayMs as 0.
+ standardScaler
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(windowSizeMs)))
+ .setModelVersionCol("modelVersionCol");
+ output = standardScaler.fit(inputTableWithEventTime).transform(inputTableWithEventTime)[0];
+ verifyUsedModelVersion(
Review Comment:
The prediction result is undetermined because we may use new model to predict old data.
So we can only check the model version here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1090519401
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(MLMetrics.ML_MODEL_GROUP);
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() <= modelTimeStamp + maxAllowedModelDelayMs
+ && mean != null) {
+ doPrediction(dataPoint);
+ } else {
+ bufferedPointsState.add(dataPoint);
Review Comment:
There is a bit difference between `modelDataState` and `bufferedPointsState` --- There is one single element in model data in `modelDataState` and more than one element in `bufferedPointsState`.
Appending elements into liststate is an asyncronous operation and thus can help to improve the efficiency when caching many data points. On the other side, there is only one model data. If we update it using `state::clear(), state::add()` may introduce unnessary state operations (e.g., add it to state and remove it before a snapshot).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1090519401
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(MLMetrics.ML_MODEL_GROUP);
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() <= modelTimeStamp + maxAllowedModelDelayMs
+ && mean != null) {
+ doPrediction(dataPoint);
+ } else {
+ bufferedPointsState.add(dataPoint);
Review Comment:
There is a bit difference between `modelDataState` and `bufferedPointsState` --- There is one single element in model data in `modelDataState` and more than one elements in `bufferedPointsState`.
Appending elements into liststate is an asyncronous operation and thus can help to improve the efficiency when caching data points. On the other side, there is only one model data and we updating it may introduce unnessary state operations (e.g., add it to state and remove it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1091380089
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/StandardScalerTest.java:
##########
@@ -206,7 +206,7 @@ public void testSaveLoadAndPredict() throws Exception {
model = TestUtils.saveAndReload(tEnv, model, tempFolder.newFolder().getAbsolutePath());
assertEquals(
- Arrays.asList("mean", "std"),
+ Arrays.asList("mean", "std", "version", "timestamp"),
Review Comment:
> What do you mean by verify model data contains those columns? Do you mean we only verify model data and leave model version and timestamp unchecked?
Yes that is what I means. IMO StandardScalerTest should verify the expected behavior of StandardScalerModel from the perspective of users who want to use StandardScalerModel. IMO users of StandardScalerModel should not need to know concepts such as model version or timestamp. Users should `OnlineStandardScaler` needs to know these information, and this behavior is covered by `OnlineStandardScalerTest`.
> I think letting StandardScalerModel have model version and timestamp is reasonable, since offline learning could be viewed as a special case of online learning
Yes we can tell users this. But for users of StandardScaler, there is no benefit to knowing this. So it seems simpler not to tell users of `StandardScalerModel` about version etc.?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1094445447
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(
+ MLMetrics.ML_MODEL_GROUP,
+ OnlineStandardScalerModel.class.getSimpleName());
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() - maxAllowedModelDelayMs <= modelTimeStamp
Review Comment:
Thanks for pointing this out!
The default value should be zero here. For better readability and also the fact that we have exposed `modelVersion` and `modelTimeStamp` to end users, I have added default values of `modelVersion` and `modelTimeStamp` as -1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on PR #196:
URL: https://github.com/apache/flink-ml/pull/196#issuecomment-1408518874
@lindong28 Thanks for the review. I have updated the PR according to your comments and closed some of the issues we both aggree on. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1090535539
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasModelVersionCol.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.StringParam;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared model version column param. */
+public interface HasModelVersionCol<T> extends WithParams<T> {
+ Param<String> MODEL_VERSION_COL =
+ new StringParam(
+ "modelVersionCol",
+ "The version of the model data that the input data is predicted with.",
+ null);
Review Comment:
I agree that it should not raise an observable performance impact here. I set the default value as `null` since it is consistent with the exiting prediction case. Is there any other benefits (except debugging) to output the model version?
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasModelVersionCol.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.StringParam;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared model version column param. */
+public interface HasModelVersionCol<T> extends WithParams<T> {
+ Param<String> MODEL_VERSION_COL =
+ new StringParam(
+ "modelVersionCol",
+ "The version of the model data that the input data is predicted with.",
+ null);
Review Comment:
I agree that it should not raise an observable performance impact here. I set the default value as `null` since it is consistent with the exiting prediction case. Is there any other benefit (except debugging) to output the model version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1068995637
##########
docs/content/docs/operators/feature/onlinestandardscaler.md:
##########
@@ -0,0 +1,261 @@
+---
+title: "OnlineStandardScaler"
+weight: 1
+type: docs
+aliases:
+- /operators/feature/onlinestandardscaler.html
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## OnlineStandardScaler
+
+An Estimator which implements the online standard scaling algorithm, which
+is the online version of StandardScaler.
+
+OnlineStandardScaler splits the input data by the user-specified window strategy.
+For each window, it computes the mean and standard deviation using the data seen
+so far (i.e., not only the data in the current window, but also the history data).
+The model data generated by OnlineStandardScaler is a model stream.
+There is one model data for each window.
+
+During the inference phase (i.e., using OnlineStandardScalerModel for prediction),
+users could output the model version that is used for predicting each data point.
+Moreover,
+- When the train data and test data both contain event time, users could
+specify the maximum difference between timestamp of the input and model data,
Review Comment:
ah, thanks for pointing this out, it is fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] jiangxin369 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by GitBox <gi...@apache.org>.
jiangxin369 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1064248240
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasMaxAllowedModelDelayMs.java:
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.LongParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared max allowed model delay in milliseconds param. */
+public interface HasMaxAllowedModelDelayMs<T> extends WithParams<T> {
+ Param<Long> MAX_ALLOWED_MODEL_DELAY_MS =
+ new LongParam(
+ "maxAllowedModelDelayMs",
+ "The maximum difference between timestamp of the input record and model data when "
Review Comment:
```suggestion
"The maximum difference between the timestamps of the input record and model data when "
```
So as in Python and Markdown files.
##########
flink-ml-python/pyflink/ml/lib/param.py:
##########
@@ -581,3 +581,48 @@ def get_flatten(self) -> bool:
@property
def flatten(self):
return self.get(self.FLATTEN)
+
+
+class HasModelVersionCol(WithParams, ABC):
Review Comment:
Could you add tests of these two params in `test_param.py`?
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScaler.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.ProcessAllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * An Estimator which implements the online standard scaling algorithm, which is the online version
+ * of {@link StandardScaler}.
+ *
+ * <p>OnlineStandardScaler splits the input data by the user-specified window strategy (i.e., {@link
+ * org.apache.flink.ml.common.param.HasWindows}). For each window, it computes the mean and standard
+ * deviation using the data seen so far (i.e., not only the data in the current window, but also the
+ * history data). The model data generated by OnlineStandardScaler is a model stream. There is one
+ * model data for each window.
+ *
+ * <p>During the inference phase (i.e., using {@link OnlineStandardScalerModel} for prediction),
+ * users could output the model version that is used for predicting each data point. Moreover,
+ *
+ * <ul>
+ * <li>When the train data and test data both contains event time, users could specify the maximum
+ * difference between timestamp of the input and model data ({@link
+ * org.apache.flink.ml.common.param.HasMaxAllowedModelDelayMs}), which enforces to use a
+ * relatively fresh model for prediction.
+ * <li>Otherwise, the prediction process always use the current model data for prediction.
+ * </ul>
+ */
+public class OnlineStandardScaler
+ implements Estimator<OnlineStandardScaler, OnlineStandardScalerModel>,
+ OnlineStandardScalerParams<OnlineStandardScaler> {
+
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public OnlineStandardScaler() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public OnlineStandardScalerModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<StandardScalerModelData> modelData =
+ DataStreamUtils.windowAllAndProcess(
+ tEnv.toDataStream(inputs[0]),
+ getWindows(),
+ new ComputeModelDataFunction<>(getInputCol()));
+
+ OnlineStandardScalerModel model =
+ new OnlineStandardScalerModel().setModelData(tEnv.fromDataStream(modelData));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ private static class ComputeModelDataFunction<W extends Window>
+ extends ProcessAllWindowFunction<Row, StandardScalerModelData, W> {
+
+ private final String inputCol;
+
+ public ComputeModelDataFunction(String inputCol) {
+ this.inputCol = inputCol;
+ }
+
+ @Override
+ public void process(
+ ProcessAllWindowFunction<Row, StandardScalerModelData, W>.Context context,
+ Iterable<Row> iterable,
+ Collector<StandardScalerModelData> collector)
+ throws Exception {
+ ListState<DenseVector> sumState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>(
+ "sumState", DenseVectorTypeInfo.INSTANCE));
+ ListState<DenseVector> squaredSumState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>(
+ "squaredSumState", DenseVectorTypeInfo.INSTANCE));
+ ListState<Long> numElementsState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>("numElementsState", Types.LONG));
+ ListState<Long> modelVersionState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>("modelVersionState", Types.LONG));
+ DenseVector sum =
+ OperatorStateUtils.getUniqueElement(sumState, "sumState").orElse(null);
+ DenseVector squaredSum =
+ OperatorStateUtils.getUniqueElement(squaredSumState, "squaredSumState")
+ .orElse(null);
+ long numElements =
+ OperatorStateUtils.getUniqueElement(numElementsState, "numElementsState")
+ .orElse(0L);
+ long modelVersion =
+ OperatorStateUtils.getUniqueElement(modelVersionState, "modelVersionState")
+ .orElse(0L);
+
+ long numElementsBefore = numElements;
+ for (Row element : iterable) {
+ Vector inputVec =
+ ((Vector) Objects.requireNonNull(element.getField(inputCol))).clone();
Review Comment:
Why do we need to clone the input vector?
##########
flink-ml-python/pyflink/ml/lib/feature/tests/test_onlinestandardscaler.py:
##########
@@ -0,0 +1,210 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from pyflink.common import Types
+from pyflink.common.time import Time, Instant
+from pyflink.java_gateway import get_gateway
+from pyflink.table import Schema
+from pyflink.table.types import DataTypes
+from pyflink.table.expressions import col
+
+from pyflink.ml.core.linalg import Vectors, DenseVectorTypeInfo
+from pyflink.ml.lib.feature.onlinestandardscaler import OnlineStandardScaler, \
+ OnlineStandardScalerModel
+from pyflink.ml.tests.test_utils import PyFlinkMLTestCase, update_existing_params
+from pyflink.ml.core.windows import GlobalWindows, EventTimeTumblingWindows
+
+
+class OnlineStandardScalerTest(PyFlinkMLTestCase):
+ def setUp(self):
+ super(OnlineStandardScalerTest, self).setUp()
+
+ dense_vector_serializer = get_gateway().jvm.org.apache.flink.table.types.logical.RawType(
+ get_gateway().jvm.org.apache.flink.ml.linalg.DenseVector(0).getClass(),
+ get_gateway().jvm.org.apache.flink.ml.linalg.typeinfo.DenseVectorSerializer()
+ ).getSerializerString()
+
+ schema = Schema.new_builder() \
+ .column("ts_in_long", DataTypes.BIGINT()) \
+ .column("ts", "TIMESTAMP_LTZ(3)") \
+ .column("input", "RAW('org.apache.flink.ml.linalg.DenseVector', '{serializer}')"
+ .format(serializer=dense_vector_serializer)) \
+ .watermark("ts", "ts - INTERVAL '1' SECOND") \
+ .build()
+
+ self.input_table = self.t_env.from_data_stream(
+ self.env.from_collection([
+ (0, Instant.of_epoch_milli(0), Vectors.dense(-2.5, 9, 1),),
+ (1000, Instant.of_epoch_milli(1000), Vectors.dense(1.4, -5, 1),),
+ (2000, Instant.of_epoch_milli(2000), Vectors.dense(2, -1, -2),),
+ (6000, Instant.of_epoch_milli(6000), Vectors.dense(0.7, 3, 1),),
+ (7000, Instant.of_epoch_milli(7000), Vectors.dense(0, 1, 1),),
+ (8000, Instant.of_epoch_milli(8000), Vectors.dense(0.5, 0, -2),),
+ (9000, Instant.of_epoch_milli(9000), Vectors.dense(0.4, 1, 1),),
+ (10000, Instant.of_epoch_milli(10000), Vectors.dense(0.3, 2, 1),),
+ (11000, Instant.of_epoch_milli(11000), Vectors.dense(0.5, 1, -2),)
+ ],
+ type_info=Types.ROW_NAMED(
+ ['ts_in_long', 'ts', 'input'],
+ [Types.LONG(), Types.INSTANT(), DenseVectorTypeInfo()])),
+ schema)
+
+ self.expected_model_data = [
+ [
+ Vectors.dense(0.3, 1, 0),
+ Vectors.dense(2.4433583, 7.2111026, 1.7320508),
+ 0,
+ 2999
+ ],
+ [
+ Vectors.dense(0.35, 1.1666667, 0),
+ Vectors.dense(1.5630099, 4.6654760, 1.5491933),
+ 1,
+ 8999
+ ],
+ [
+ Vectors.dense(0.3666667, 1.2222222, 0),
+ Vectors.dense(1.2369316, 3.7006005, 1.5),
+ 2,
+ 11999
+ ]
+ ]
+
+ self.tolerance = 1e-7
+
+ def test_param(self):
+ standard_scaler = OnlineStandardScaler()
+
+ self.assertEqual('input', standard_scaler.input_col)
+ self.assertEqual(False, standard_scaler.with_mean)
+ self.assertEqual(True, standard_scaler.with_std)
+ self.assertEqual('output', standard_scaler.output_col)
+ self.assertIsNone(standard_scaler.model_version_col)
+ self.assertEqual(GlobalWindows(), standard_scaler.windows)
+ self.assertEqual(0, standard_scaler.max_allowed_model_delay_ms)
+
+ standard_scaler.set_input_col('test_input') \
+ .set_with_mean(True) \
+ .set_with_std(False) \
+ .set_output_col('test_output') \
+ .set_model_version_col('model_version_col') \
+ .set_windows(EventTimeTumblingWindows.of(Time.milliseconds(3000))) \
+ .set_max_allowed_model_delay_ms(3000)
+
+ self.assertEqual('test_input', standard_scaler.input_col)
+ self.assertEqual(True, standard_scaler.with_mean)
+ self.assertEqual(False, standard_scaler.with_std)
+ self.assertEqual('test_output', standard_scaler.output_col)
+ self.assertEqual("model_version_col", standard_scaler.model_version_col)
+ self.assertEqual(EventTimeTumblingWindows.of(Time.milliseconds(3000)),
+ standard_scaler.windows)
+ self.assertEqual(3000, standard_scaler.max_allowed_model_delay_ms)
+
+ def test_output_schema(self):
+ temp_table = self.input_table.alias('ts_in_long', 'ts', "test_input")
+
+ standard_scaler = OnlineStandardScaler() \
+ .set_input_col('test_input') \
+ .set_output_col('test_output') \
+ .set_model_version_col('model_version_col')
+
+ output = standard_scaler.fit(temp_table).transform(temp_table)[0]
+
+ self.assertEqual(['ts_in_long', 'ts', 'test_input', 'test_output', 'model_version_col'],
+ output.get_schema().get_field_names())
+
+ def test_fit_and_predict(self):
+ standard_scaler = OnlineStandardScaler()
+ window_size_ms = 3000
+
+ standard_scaler \
+ .set_windows(EventTimeTumblingWindows.of(Time.milliseconds(window_size_ms))) \
+ .set_model_version_col("model_version_col")
+ output = standard_scaler.fit(self.input_table).transform(self.input_table)[0]
+ self.verify_used_model_version(output, standard_scaler.model_version_col,
+ standard_scaler.max_allowed_model_delay_ms)
+
+ def test_get_model_data(self):
+ standard_scaler = OnlineStandardScaler()
+ window_size_ms = 3000
+
+ standard_scaler \
+ .set_windows(EventTimeTumblingWindows.of(Time.milliseconds(window_size_ms))) \
+ .set_model_version_col("model_version_col")
+
+ model_data = standard_scaler.fit(self.input_table).get_model_data()[0]
+ self.assertEqual(["mean", "std", "version", "timestamp"],
+ model_data.get_schema().get_field_names())
+
+ model_rows = [result for result in
+ self.t_env.to_data_stream(model_data).execute_and_collect()]
+ self.assertEqual(len(self.expected_model_data), len(model_rows))
+ for idx in range(len(self.expected_model_data)):
+ self.assertListAlmostEqual(self.expected_model_data[idx][0].to_array(),
+ model_rows[idx][0].to_array(),
+ delta=self.tolerance)
+ self.assertListAlmostEqual(self.expected_model_data[idx][1].to_array(),
+ model_rows[idx][1].to_array(),
+ delta=self.tolerance)
+ self.assertEqual(self.expected_model_data[idx][2], model_rows[idx][2])
+ self.assertEqual(self.expected_model_data[idx][3], model_rows[idx][3])
+
+ def test_set_model_data(self):
+ standard_scaler = OnlineStandardScaler()
+ window_size_ms = 3000
+
+ model = standard_scaler \
+ .set_windows(EventTimeTumblingWindows.of(Time.milliseconds(window_size_ms))) \
+ .set_model_version_col("model_version_col") \
+ .fit(self.input_table)
+ model_data = model.get_model_data()[0]
+
+ new_model = OnlineStandardScalerModel().set_model_data(model_data)
+ update_existing_params(new_model, model)
+ output = new_model.transform(self.input_table)[0]
+
+ self.verify_used_model_version(output, model.model_version_col,
+ model.max_allowed_model_delay_ms)
+
+ def test_save_load_and_predict(self):
+ window_size_ms = 3000
Review Comment:
Would it be better if declare the `window_size_ms` as a shared constant?
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/OnlineStandardScalerTest.java:
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature;
+
+import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.common.window.CountTumblingWindows;
+import org.apache.flink.ml.common.window.EventTimeTumblingWindows;
+import org.apache.flink.ml.common.window.GlobalWindows;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScaler;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScalerModel;
+import org.apache.flink.ml.feature.standardscaler.StandardScalerModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.TestUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.test.util.AbstractTestBase;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests {@link OnlineStandardScaler} and {@link OnlineStandardScalerModel}. */
+public class OnlineStandardScalerTest extends AbstractTestBase {
+ @Rule public final TemporaryFolder tempFolder = new TemporaryFolder();
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private final List<Row> inputData =
+ Arrays.asList(
+ Row.of(0L, Vectors.dense(-2.5, 9, 1)),
+ Row.of(1000L, Vectors.dense(1.4, -5, 1)),
+ Row.of(2000L, Vectors.dense(2, -1, -2)),
+ Row.of(6000L, Vectors.dense(0.7, 3, 1)),
+ Row.of(7000L, Vectors.dense(0, 1, 1)),
+ Row.of(8000L, Vectors.dense(0.5, 0, -2)),
+ Row.of(9000L, Vectors.dense(0.4, 1, 1)),
+ Row.of(10000L, Vectors.dense(0.3, 2, 1)),
+ Row.of(11000L, Vectors.dense(0.5, 1, -2)));
+
+ private final List<StandardScalerModelData> expectedModelData =
+ Arrays.asList(
+ new StandardScalerModelData(
+ Vectors.dense(0.3, 1, 0),
+ Vectors.dense(2.4433583, 7.2111026, 1.7320508),
+ 0L,
+ 2999L),
+ new StandardScalerModelData(
+ Vectors.dense(0.35, 1.1666667, 0),
+ Vectors.dense(1.5630099, 4.6654760, 1.5491933),
+ 1L,
+ 8999L),
+ new StandardScalerModelData(
+ Vectors.dense(0.3666667, 1.2222222, 0),
+ Vectors.dense(1.2369316, 3.7006005, 1.5),
+ 2L,
+ 11999L));
+
+ private static final double TOLERANCE = 1e-7;
+
+ private Table inputTable;
+
+ private Table inputTableWithEventTime;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.getConfig().enableObjectReuse();
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ DataStream<Row> inputStream = env.fromCollection(inputData);
+ inputTable =
+ tEnv.fromDataStream(
+ inputStream,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .build())
+ .as("id", "input");
+
+ DataStream<Row> inputStreamWithEventTime =
+ inputStream.assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Row>forMonotonousTimestamps()
+ .withTimestampAssigner(
+ (SerializableTimestampAssigner<Row>)
+ (element, recordTimestamp) ->
+ element.getFieldAs(0)));
+ inputTableWithEventTime =
+ tEnv.fromDataStream(
+ inputStreamWithEventTime,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build())
+ .as("id", "input");
+ }
+
+ @Test
+ public void testParam() {
+ OnlineStandardScaler standardScaler = new OnlineStandardScaler();
+
+ assertEquals("input", standardScaler.getInputCol());
+ assertEquals(false, standardScaler.getWithMean());
+ assertEquals(true, standardScaler.getWithStd());
+ assertEquals("output", standardScaler.getOutputCol());
+ assertNull(standardScaler.getModelVersionCol());
+ assertEquals(GlobalWindows.getInstance(), standardScaler.getWindows());
+ assertEquals(0L, standardScaler.getMaxAllowedModelDelayMs());
+
+ standardScaler
+ .setInputCol("test_input")
+ .setWithMean(true)
+ .setWithStd(false)
+ .setOutputCol("test_output")
+ .setModelVersionCol("model_version_col")
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(3000)))
+ .setMaxAllowedModelDelayMs(3000L);
+
+ assertEquals("test_input", standardScaler.getInputCol());
+ assertEquals(true, standardScaler.getWithMean());
+ assertEquals(false, standardScaler.getWithStd());
+ assertEquals("test_output", standardScaler.getOutputCol());
+ assertEquals("model_version_col", standardScaler.getModelVersionCol());
+ assertEquals(
+ EventTimeTumblingWindows.of(Time.milliseconds(3000)), standardScaler.getWindows());
+ assertEquals(3000L, standardScaler.getMaxAllowedModelDelayMs());
+ }
+
+ @Test
+ public void testOutputSchema() {
+ Table renamedTable = inputTable.as("test_id", "test_input");
+ OnlineStandardScaler standardScaler =
+ new OnlineStandardScaler()
+ .setInputCol("test_input")
+ .setOutputCol("test_output")
+ .setModelVersionCol("model_version_col");
+ Table output = standardScaler.fit(renamedTable).transform(renamedTable)[0];
+
+ assertEquals(
+ Arrays.asList("test_id", "test_input", "test_output", "model_version_col"),
+ output.getResolvedSchema().getColumnNames());
+ }
+
+ @Test
+ public void testFitAndPredictWithEventTimeWindow() throws Exception {
+ OnlineStandardScaler standardScaler = new OnlineStandardScaler();
+ Table output;
+ int windowSizeMs = 3000;
+
+ // Tests event time window with maxAllowedModelDelayMs as 0.
+ standardScaler
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(windowSizeMs)))
+ .setModelVersionCol("modelVersionCol");
+ output = standardScaler.fit(inputTableWithEventTime).transform(inputTableWithEventTime)[0];
+ verifyUsedModelVersion(
Review Comment:
Could you also verify the outputs of the prediction?
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerParams.java:
##########
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.ml.common.param.HasWindows;
+
+/** Params for {@link OnlineStandardScaler}. */
+public interface OnlineStandardScalerParams<T>
+ extends HasWindows<T>, OnlineStandardScalerModelParams<T> {}
Review Comment:
From my testing, not all `Windows` are supported, e.g., when using `ProcessingTimeTumblingWindows`, the `OnlineStandardScaler` cannot train any models. Is that expected behavior? If so, maybe we can add a validator and document the windows we supported.
##########
docs/content/docs/operators/feature/onlinestandardscaler.md:
##########
@@ -0,0 +1,261 @@
+---
+title: "OnlineStandardScaler"
+weight: 1
+type: docs
+aliases:
+- /operators/feature/onlinestandardscaler.html
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## OnlineStandardScaler
+
+An Estimator which implements the online standard scaling algorithm, which
+is the online version of StandardScaler.
+
+OnlineStandardScaler splits the input data by the user-specified window strategy.
+For each window, it computes the mean and standard deviation using the data seen
+so far (i.e., not only the data in the current window, but also the history data).
+The model data generated by OnlineStandardScaler is a model stream.
+There is one model data for each window.
+
+During the inference phase (i.e., using OnlineStandardScalerModel for prediction),
+users could output the model version that is used for predicting each data point.
+Moreover,
+- When the train data and test data both contains event time, users could
+specify the maximum difference between timestamp of the input and model data,
+which enforces to use a relatively fresh model for prediction.
+- Otherwise, the prediction process always use the current model data for prediction.
Review Comment:
```suggestion
- Otherwise, the prediction process always uses the current model data for prediction.
```
##########
docs/content/docs/operators/feature/onlinestandardscaler.md:
##########
@@ -0,0 +1,261 @@
+---
+title: "OnlineStandardScaler"
+weight: 1
+type: docs
+aliases:
+- /operators/feature/onlinestandardscaler.html
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## OnlineStandardScaler
+
+An Estimator which implements the online standard scaling algorithm, which
+is the online version of StandardScaler.
+
+OnlineStandardScaler splits the input data by the user-specified window strategy.
+For each window, it computes the mean and standard deviation using the data seen
+so far (i.e., not only the data in the current window, but also the history data).
+The model data generated by OnlineStandardScaler is a model stream.
+There is one model data for each window.
+
+During the inference phase (i.e., using OnlineStandardScalerModel for prediction),
+users could output the model version that is used for predicting each data point.
+Moreover,
+- When the train data and test data both contains event time, users could
Review Comment:
```suggestion
- When the train data and test data both contain event time, users could
```
So as in Javadoc and Python.
##########
docs/content/docs/operators/feature/onlinestandardscaler.md:
##########
@@ -0,0 +1,261 @@
+---
+title: "OnlineStandardScaler"
+weight: 1
+type: docs
+aliases:
+- /operators/feature/onlinestandardscaler.html
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## OnlineStandardScaler
+
+An Estimator which implements the online standard scaling algorithm, which
+is the online version of StandardScaler.
+
+OnlineStandardScaler splits the input data by the user-specified window strategy.
+For each window, it computes the mean and standard deviation using the data seen
+so far (i.e., not only the data in the current window, but also the history data).
+The model data generated by OnlineStandardScaler is a model stream.
+There is one model data for each window.
+
+During the inference phase (i.e., using OnlineStandardScalerModel for prediction),
+users could output the model version that is used for predicting each data point.
+Moreover,
+- When the train data and test data both contains event time, users could
+specify the maximum difference between timestamp of the input and model data,
+which enforces to use a relatively fresh model for prediction.
+- Otherwise, the prediction process always use the current model data for prediction.
+
+
+### Input Columns
+
+| Param name | Type | Default | Description |
+|:-----------|:-------|:----------|:-----------------------|
+| inputCol | Vector | `"input"` | Features to be scaled. |
+
+### Output Columns
+
+| Param name | Type | Default | Description |
+|:-----------|:-------|:-----------|:-----------------|
+| outputCol | Vector | `"output"` | Scaled features. |
+
+### Parameters
+
+Below are the parameters required by `OnlineStandardScalerModel`.
+
+| Key | Default | Type | Required | Description |
+|------------------------|------------|---------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| inputCol | `"input"` | String | no | Input column name. |
+| outputCol | `"output"` | String | no | Output column name. |
+| withMean | `false` | Boolean | no | Whether centers the data with mean before scaling. |
+| withStd | `true` | Boolean | no | Whether scales the data with standard deviation. |
+ | modelVersionCol | `null` | String | no | The version of the model data that the input data is predicted with. |
+ | maxAllowedModelDelayMs | `0L` | Long | no | The maximum difference between timestamp of the input record and model data when using the model data to predict that input record. This param only works when the input contains event time." |
+
+`OnlineStandardScaler` needs parameters above and also below.
+
+| Key | Default | Type | Required | Description |
+|---------|-------------------------------|---------|----------|--------------------------------------------------------------------------------|
+| windows | `GlobalWindows.getInstance()` | Windows | no | Windowing strategy that determines how to create mini-batches from input data. |
+
+
+### Examples
+
+{{< tabs examples >}}
+
+{{< tab "Java">}}
+
+```java
+import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.ml.common.window.EventTimeTumblingWindows;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScaler;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScalerModel;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import java.util.Arrays;
+import java.util.List;
+
+/** Simple program that trains a OnlineStandardScaler model and uses it for feature engineering. */
+public class OnlineStandardScalerExample {
+ public static void main(String[] args) {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ // Generates input data.
+ List<Row> inputData =
+ Arrays.asList(
+ Row.of(0L, Vectors.dense(-2.5, 9, 1)),
+ Row.of(1000L, Vectors.dense(1.4, -5, 1)),
+ Row.of(2000L, Vectors.dense(2, -1, -2)),
+ Row.of(6000L, Vectors.dense(0.7, 3, 1)),
+ Row.of(7000L, Vectors.dense(0, 1, 1)),
+ Row.of(8000L, Vectors.dense(0.5, 0, -2)),
+ Row.of(9000L, Vectors.dense(0.4, 1, 1)),
+ Row.of(10000L, Vectors.dense(0.3, 2, 1)),
+ Row.of(11000L, Vectors.dense(0.5, 1, -2)));
+
+ DataStream<Row> inputStream = env.fromCollection(inputData);
+
+ DataStream<Row> inputStreamWithEventTime =
+ inputStream.assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Row>forMonotonousTimestamps()
+ .withTimestampAssigner(
+ (SerializableTimestampAssigner<Row>)
+ (element, recordTimestamp) ->
+ element.getFieldAs(0)));
+
+ Table inputTable =
+ tEnv.fromDataStream(
+ inputStreamWithEventTime,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build())
+ .as("id", "input");
+
+ // Creates an OnlineStandardScaler object and initializes its parameters.
+ long windowSizeMs = 3000;
+ OnlineStandardScaler onlineStandardScaler =
+ new OnlineStandardScaler()
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(windowSizeMs)))
+ .setModelVersionCol("modelVersionCol");
+
+ // Trains the OnlineStandardScaler Model.
+ OnlineStandardScalerModel model = onlineStandardScaler.fit(inputTable);
+
+ // Uses the OnlineStandardScaler Model for predictions.
+ Table outputTable = model.transform(inputTable)[0];
+
+ // Extracts and displays the results.
+ for (CloseableIterator<Row> it = outputTable.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ DenseVector inputValue = (DenseVector) row.getField(onlineStandardScaler.getInputCol());
+ DenseVector outputValue =
+ (DenseVector) row.getField(onlineStandardScaler.getOutputCol());
+ long modelVersion = row.getFieldAs(onlineStandardScaler.getModelVersionCol());
+ System.out.printf(
+ "Input Value: %s\tOutput Value: %s\tModel Version: %s\n",
+ inputValue, outputValue, modelVersion);
+ }
+ }
+}
+
Review Comment:
Please remove the redundant empty line.
##########
docs/content/docs/operators/feature/onlinestandardscaler.md:
##########
@@ -0,0 +1,261 @@
+---
+title: "OnlineStandardScaler"
+weight: 1
+type: docs
+aliases:
+- /operators/feature/onlinestandardscaler.html
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## OnlineStandardScaler
+
+An Estimator which implements the online standard scaling algorithm, which
+is the online version of StandardScaler.
+
+OnlineStandardScaler splits the input data by the user-specified window strategy.
+For each window, it computes the mean and standard deviation using the data seen
+so far (i.e., not only the data in the current window, but also the history data).
+The model data generated by OnlineStandardScaler is a model stream.
+There is one model data for each window.
+
+During the inference phase (i.e., using OnlineStandardScalerModel for prediction),
+users could output the model version that is used for predicting each data point.
+Moreover,
+- When the train data and test data both contains event time, users could
+specify the maximum difference between timestamp of the input and model data,
+which enforces to use a relatively fresh model for prediction.
+- Otherwise, the prediction process always use the current model data for prediction.
+
+
+### Input Columns
+
+| Param name | Type | Default | Description |
+|:-----------|:-------|:----------|:-----------------------|
+| inputCol | Vector | `"input"` | Features to be scaled. |
+
+### Output Columns
+
+| Param name | Type | Default | Description |
+|:-----------|:-------|:-----------|:-----------------|
+| outputCol | Vector | `"output"` | Scaled features. |
+
+### Parameters
+
+Below are the parameters required by `OnlineStandardScalerModel`.
+
+| Key | Default | Type | Required | Description |
+|------------------------|------------|---------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| inputCol | `"input"` | String | no | Input column name. |
+| outputCol | `"output"` | String | no | Output column name. |
+| withMean | `false` | Boolean | no | Whether centers the data with mean before scaling. |
+| withStd | `true` | Boolean | no | Whether scales the data with standard deviation. |
+ | modelVersionCol | `null` | String | no | The version of the model data that the input data is predicted with. |
Review Comment:
```suggestion
| modelVersionCol | `null` | String | no | The version of the model data that the input data is predicted with. |
```
So as the below line.
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScaler.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.ProcessAllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * An Estimator which implements the online standard scaling algorithm, which is the online version
+ * of {@link StandardScaler}.
+ *
+ * <p>OnlineStandardScaler splits the input data by the user-specified window strategy (i.e., {@link
+ * org.apache.flink.ml.common.param.HasWindows}). For each window, it computes the mean and standard
+ * deviation using the data seen so far (i.e., not only the data in the current window, but also the
+ * history data). The model data generated by OnlineStandardScaler is a model stream. There is one
+ * model data for each window.
+ *
+ * <p>During the inference phase (i.e., using {@link OnlineStandardScalerModel} for prediction),
+ * users could output the model version that is used for predicting each data point. Moreover,
+ *
+ * <ul>
+ * <li>When the train data and test data both contains event time, users could specify the maximum
+ * difference between timestamp of the input and model data ({@link
+ * org.apache.flink.ml.common.param.HasMaxAllowedModelDelayMs}), which enforces to use a
+ * relatively fresh model for prediction.
+ * <li>Otherwise, the prediction process always use the current model data for prediction.
+ * </ul>
+ */
+public class OnlineStandardScaler
+ implements Estimator<OnlineStandardScaler, OnlineStandardScalerModel>,
+ OnlineStandardScalerParams<OnlineStandardScaler> {
+
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public OnlineStandardScaler() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public OnlineStandardScalerModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<StandardScalerModelData> modelData =
+ DataStreamUtils.windowAllAndProcess(
+ tEnv.toDataStream(inputs[0]),
+ getWindows(),
+ new ComputeModelDataFunction<>(getInputCol()));
+
+ OnlineStandardScalerModel model =
+ new OnlineStandardScalerModel().setModelData(tEnv.fromDataStream(modelData));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ private static class ComputeModelDataFunction<W extends Window>
+ extends ProcessAllWindowFunction<Row, StandardScalerModelData, W> {
+
+ private final String inputCol;
+
+ public ComputeModelDataFunction(String inputCol) {
+ this.inputCol = inputCol;
+ }
+
+ @Override
+ public void process(
+ ProcessAllWindowFunction<Row, StandardScalerModelData, W>.Context context,
+ Iterable<Row> iterable,
+ Collector<StandardScalerModelData> collector)
+ throws Exception {
+ ListState<DenseVector> sumState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>(
+ "sumState", DenseVectorTypeInfo.INSTANCE));
+ ListState<DenseVector> squaredSumState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>(
+ "squaredSumState", DenseVectorTypeInfo.INSTANCE));
+ ListState<Long> numElementsState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>("numElementsState", Types.LONG));
+ ListState<Long> modelVersionState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>("modelVersionState", Types.LONG));
+ DenseVector sum =
+ OperatorStateUtils.getUniqueElement(sumState, "sumState").orElse(null);
+ DenseVector squaredSum =
+ OperatorStateUtils.getUniqueElement(squaredSumState, "squaredSumState")
+ .orElse(null);
+ long numElements =
+ OperatorStateUtils.getUniqueElement(numElementsState, "numElementsState")
+ .orElse(0L);
+ long modelVersion =
+ OperatorStateUtils.getUniqueElement(modelVersionState, "modelVersionState")
+ .orElse(0L);
+
+ long numElementsBefore = numElements;
+ for (Row element : iterable) {
+ Vector inputVec =
+ ((Vector) Objects.requireNonNull(element.getField(inputCol))).clone();
+ if (numElements == 0) {
Review Comment:
How about moving the `if` block out of the `for` loop?
##########
flink-ml-examples/src/main/java/org/apache/flink/ml/examples/feature/OnlineStandardScalerExample.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.examples.feature;
+
+import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.ml.common.window.EventTimeTumblingWindows;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScaler;
+import org.apache.flink.ml.feature.standardscaler.OnlineStandardScalerModel;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import java.util.Arrays;
+import java.util.List;
+
+/** Simple program that trains a OnlineStandardScaler model and uses it for feature engineering. */
+public class OnlineStandardScalerExample {
+ public static void main(String[] args) {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ // Generates input data.
+ List<Row> inputData =
+ Arrays.asList(
+ Row.of(0L, Vectors.dense(-2.5, 9, 1)),
+ Row.of(1000L, Vectors.dense(1.4, -5, 1)),
+ Row.of(2000L, Vectors.dense(2, -1, -2)),
+ Row.of(6000L, Vectors.dense(0.7, 3, 1)),
+ Row.of(7000L, Vectors.dense(0, 1, 1)),
+ Row.of(8000L, Vectors.dense(0.5, 0, -2)),
+ Row.of(9000L, Vectors.dense(0.4, 1, 1)),
+ Row.of(10000L, Vectors.dense(0.3, 2, 1)),
+ Row.of(11000L, Vectors.dense(0.5, 1, -2)));
+
+ DataStream<Row> inputStream = env.fromCollection(inputData);
+
+ DataStream<Row> inputStreamWithEventTime =
+ inputStream.assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Row>forMonotonousTimestamps()
+ .withTimestampAssigner(
+ (SerializableTimestampAssigner<Row>)
+ (element, recordTimestamp) ->
+ element.getFieldAs(0)));
+
+ Table inputTable =
+ tEnv.fromDataStream(
+ inputStreamWithEventTime,
+ Schema.newBuilder()
+ .column("f0", DataTypes.BIGINT())
+ .column("f1", DataTypes.RAW(DenseVectorTypeInfo.INSTANCE))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build())
+ .as("id", "input");
+
+ // Creates an OnlineStandardScaler object and initializes its parameters.
+ long windowSizeMs = 3000;
+ OnlineStandardScaler onlineStandardScaler =
+ new OnlineStandardScaler()
+ .setWindows(EventTimeTumblingWindows.of(Time.milliseconds(windowSizeMs)))
+ .setModelVersionCol("modelVersionCol");
+
+ // Trains the OnlineStandardScaler Model.
+ OnlineStandardScalerModel model = onlineStandardScaler.fit(inputTable);
+
+ // Uses the OnlineStandardScaler Model for predictions.
+ Table outputTable = model.transform(inputTable)[0];
+
+ // Extracts and displays the results.
+ for (CloseableIterator<Row> it = outputTable.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ DenseVector inputValue = (DenseVector) row.getField(onlineStandardScaler.getInputCol());
+ DenseVector outputValue =
+ (DenseVector) row.getField(onlineStandardScaler.getOutputCol());
+ long modelVersion = row.getFieldAs(onlineStandardScaler.getModelVersionCol());
+ System.out.printf(
+ "Input Value: %s\tOutput Value: %s\tModel Version: %s\n",
Review Comment:
Would it be better if we align the output of the `Model Version` column? E.g., `%-65s\t`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1066574806
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScaler.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.ProcessAllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * An Estimator which implements the online standard scaling algorithm, which is the online version
+ * of {@link StandardScaler}.
+ *
+ * <p>OnlineStandardScaler splits the input data by the user-specified window strategy (i.e., {@link
+ * org.apache.flink.ml.common.param.HasWindows}). For each window, it computes the mean and standard
+ * deviation using the data seen so far (i.e., not only the data in the current window, but also the
+ * history data). The model data generated by OnlineStandardScaler is a model stream. There is one
+ * model data for each window.
+ *
+ * <p>During the inference phase (i.e., using {@link OnlineStandardScalerModel} for prediction),
+ * users could output the model version that is used for predicting each data point. Moreover,
+ *
+ * <ul>
+ * <li>When the train data and test data both contains event time, users could specify the maximum
+ * difference between timestamp of the input and model data ({@link
+ * org.apache.flink.ml.common.param.HasMaxAllowedModelDelayMs}), which enforces to use a
+ * relatively fresh model for prediction.
+ * <li>Otherwise, the prediction process always use the current model data for prediction.
+ * </ul>
+ */
+public class OnlineStandardScaler
+ implements Estimator<OnlineStandardScaler, OnlineStandardScalerModel>,
+ OnlineStandardScalerParams<OnlineStandardScaler> {
+
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public OnlineStandardScaler() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public OnlineStandardScalerModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<StandardScalerModelData> modelData =
+ DataStreamUtils.windowAllAndProcess(
+ tEnv.toDataStream(inputs[0]),
+ getWindows(),
+ new ComputeModelDataFunction<>(getInputCol()));
+
+ OnlineStandardScalerModel model =
+ new OnlineStandardScalerModel().setModelData(tEnv.fromDataStream(modelData));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ private static class ComputeModelDataFunction<W extends Window>
+ extends ProcessAllWindowFunction<Row, StandardScalerModelData, W> {
+
+ private final String inputCol;
+
+ public ComputeModelDataFunction(String inputCol) {
+ this.inputCol = inputCol;
+ }
+
+ @Override
+ public void process(
+ ProcessAllWindowFunction<Row, StandardScalerModelData, W>.Context context,
+ Iterable<Row> iterable,
+ Collector<StandardScalerModelData> collector)
+ throws Exception {
+ ListState<DenseVector> sumState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>(
+ "sumState", DenseVectorTypeInfo.INSTANCE));
+ ListState<DenseVector> squaredSumState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>(
+ "squaredSumState", DenseVectorTypeInfo.INSTANCE));
+ ListState<Long> numElementsState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>("numElementsState", Types.LONG));
+ ListState<Long> modelVersionState =
+ context.globalState()
+ .getListState(
+ new ListStateDescriptor<>("modelVersionState", Types.LONG));
+ DenseVector sum =
+ OperatorStateUtils.getUniqueElement(sumState, "sumState").orElse(null);
+ DenseVector squaredSum =
+ OperatorStateUtils.getUniqueElement(squaredSumState, "squaredSumState")
+ .orElse(null);
+ long numElements =
+ OperatorStateUtils.getUniqueElement(numElementsState, "numElementsState")
+ .orElse(0L);
+ long modelVersion =
+ OperatorStateUtils.getUniqueElement(modelVersionState, "modelVersionState")
+ .orElse(0L);
+
+ long numElementsBefore = numElements;
+ for (Row element : iterable) {
+ Vector inputVec =
+ ((Vector) Objects.requireNonNull(element.getField(inputCol))).clone();
+ if (numElements == 0) {
Review Comment:
Puting it inside the loop is to get the size of the vector for initializing `sum` and `squareSum`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1066632174
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerParams.java:
##########
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.ml.common.param.HasWindows;
+
+/** Params for {@link OnlineStandardScaler}. */
+public interface OnlineStandardScalerParams<T>
+ extends HasWindows<T>, OnlineStandardScalerModelParams<T> {}
Review Comment:
It supports `ProcessintTimeTumbingWindows` and I have added a corresponding unit test. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1090392883
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(MLMetrics.ML_MODEL_GROUP);
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() <= modelTimeStamp + maxAllowedModelDelayMs
Review Comment:
Good catch, it is indeed a bug. I have fixed the training logic as follows:
- When the window strategy is based on event-time, the timestamp of model data is the maximum timestamp of the window that generates this model.
- Otherwise, the timestamp of each model data is `LONG.MAX_VALUE`.
Moreover, to avoid the possible overflow, the `if condition` here is updated as `if (dataPoint.getTimestamp() - maxAllowedModelDelayMs <= modelTimeStamp`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1090525929
##########
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/StandardScalerTest.java:
##########
@@ -206,7 +206,7 @@ public void testSaveLoadAndPredict() throws Exception {
model = TestUtils.saveAndReload(tEnv, model, tempFolder.newFolder().getAbsolutePath());
assertEquals(
- Arrays.asList("mean", "std"),
+ Arrays.asList("mean", "std", "version", "timestamp"),
Review Comment:
Well, I think letting `StandardScalerModel` have model version and timestamp is reasonable, since offline learning could be viewed as a special case of online learning. What we need to do is to make sure that version and timestamp has a clear meaning. For example, the version of offline model is always zero and timestamp is `LONG.MAX_VALUE`.
What do you mean by verify `model data contains those columns`? Do you mean we only verify `model data` and leave `model version` and `timestamp` unchecked?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 merged pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 merged PR #196:
URL: https://github.com/apache/flink-ml/pull/196
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1091387514
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(MLMetrics.ML_MODEL_GROUP);
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() <= modelTimeStamp + maxAllowedModelDelayMs
+ && mean != null) {
+ doPrediction(dataPoint);
+ } else {
+ bufferedPointsState.add(dataPoint);
Review Comment:
Thanks for the explanation. Sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1091394858
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasModelVersionCol.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.StringParam;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared model version column param. */
+public interface HasModelVersionCol<T> extends WithParams<T> {
+ Param<String> MODEL_VERSION_COL =
+ new StringParam(
+ "modelVersionCol",
+ "The version of the model data that the input data is predicted with.",
+ null);
Review Comment:
By `the exiting prediction case`, do you mean `HasPredictionCol`? Currently `HasPredictionCol` defaults to prediction. Can you clarify it here?
The benefit is to minimize the config that users need to tune for the common scenario targeted by algorithms that depends on `HasModelVersionCol`.
I suppose that for algorithms such as `OnlineStandardScaler`, their users will run the algorithm continuously and observe multiple models emitted by it. And it seems useful for users to know "which version of the model data that the input data is predicted with". Since there is observable usability benefit and negligible performance cost, it seems better to enable this column by default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1092940240
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasModelVersionCol.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.StringParam;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared model version column param. */
+public interface HasModelVersionCol<T> extends WithParams<T> {
+ Param<String> MODEL_VERSION_COL =
+ new StringParam(
+ "modelVersionCol",
+ "The version of the model data that the input data is predicted with.",
+ null);
Review Comment:
Thanks for the explaination. I have changed the default value of `HasModelVersionCol` as `version`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "zhipeng93 (via GitHub)" <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1094413693
##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/standardscaler/OnlineStandardScalerModel.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.standardscaler;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.common.metrics.MLMetrics;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.typeinfo.VectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A Model which transforms data using the model data computed by {@link OnlineStandardScaler}. */
+public class OnlineStandardScalerModel
+ implements Model<OnlineStandardScalerModel>,
+ OnlineStandardScalerModelParams<OnlineStandardScalerModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public OnlineStandardScalerModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ String modelVersionCol = getModelVersionCol();
+
+ TypeInformation<?>[] outputTypes;
+ String[] outputNames;
+ if (modelVersionCol == null) {
+ outputTypes = ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE);
+ outputNames = ArrayUtils.addAll(inputTypeInfo.getFieldNames(), getOutputCol());
+ } else {
+ outputTypes =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), VectorTypeInfo.INSTANCE, Types.LONG);
+ outputNames =
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldNames(), getOutputCol(), modelVersionCol);
+ }
+ RowTypeInfo outputTypeInfo = new RowTypeInfo(outputTypes, outputNames);
+
+ DataStream<Row> predictionResult =
+ tEnv.toDataStream(inputs[0])
+ .connect(
+ StandardScalerModelData.getModelDataStream(modelDataTable)
+ .broadcast())
+ .transform(
+ "PredictionOperator",
+ outputTypeInfo,
+ new PredictionOperator(
+ inputTypeInfo,
+ getInputCol(),
+ getWithMean(),
+ getWithStd(),
+ getMaxAllowedModelDelayMs(),
+ getModelVersionCol()));
+
+ return new Table[] {tEnv.fromDataStream(predictionResult)};
+ }
+
+ /** A utility operator used for prediction. */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static class PredictionOperator extends AbstractStreamOperator<Row>
+ implements TwoInputStreamOperator<Row, StandardScalerModelData, Row> {
+ private final RowTypeInfo inputTypeInfo;
+
+ private final String inputCol;
+
+ private final boolean withMean;
+
+ private final boolean withStd;
+
+ private final long maxAllowedModelDelayMs;
+
+ private final String modelVersionCol;
+
+ private ListState<StreamRecord> bufferedPointsState;
+
+ private ListState<StandardScalerModelData> modelDataState;
+
+ /** Model data for inference. */
+ private StandardScalerModelData modelData;
+
+ private DenseVector mean;
+
+ /** Inverse of standard deviation. */
+ private DenseVector scale;
+
+ private long modelVersion;
+
+ private long modelTimeStamp;
+
+ public PredictionOperator(
+ RowTypeInfo inputTypeInfo,
+ String inputCol,
+ boolean withMean,
+ boolean withStd,
+ long maxAllowedModelDelayMs,
+ String modelVersionCol) {
+ this.inputTypeInfo = inputTypeInfo;
+ this.inputCol = inputCol;
+ this.withMean = withMean;
+ this.withStd = withStd;
+ this.maxAllowedModelDelayMs = maxAllowedModelDelayMs;
+ this.modelVersionCol = modelVersionCol;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ bufferedPointsState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<StreamRecord>(
+ "bufferedPoints",
+ new StreamElementSerializer(
+ inputTypeInfo.createSerializer(
+ getExecutionConfig()))));
+
+ modelDataState =
+ context.getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "modelData",
+ TypeInformation.of(StandardScalerModelData.class)));
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ if (modelData != null) {
+ modelDataState.clear();
+ modelDataState.add(modelData);
+ }
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mlModelMetricGroup =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MLMetrics.ML_GROUP)
+ .addGroup(
+ MLMetrics.ML_MODEL_GROUP,
+ OnlineStandardScalerModel.class.getSimpleName());
+ mlModelMetricGroup.gauge(MLMetrics.TIMESTAMP, (Gauge<Long>) () -> modelTimeStamp);
+ mlModelMetricGroup.gauge(MLMetrics.VERSION, (Gauge<Long>) () -> modelVersion);
+ }
+
+ @Override
+ public void processElement1(StreamRecord<Row> dataPoint) throws Exception {
+ if (dataPoint.getTimestamp() - maxAllowedModelDelayMs <= modelTimeStamp
+ && mean != null) {
Review Comment:
Yes.
We probably still need the check since the timestamp of the data point could also be `LONG.MIN_VALUE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on PR #196:
URL: https://github.com/apache/flink-ml/pull/196#issuecomment-1413823466
Thanks for the update! LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on code in PR #196:
URL: https://github.com/apache/flink-ml/pull/196#discussion_r1066549248
##########
flink-ml-python/pyflink/ml/lib/param.py:
##########
@@ -581,3 +581,48 @@ def get_flatten(self) -> bool:
@property
def flatten(self):
return self.get(self.FLATTEN)
+
+
+class HasModelVersionCol(WithParams, ABC):
Review Comment:
Good catch. I will add the test there.
##########
flink-ml-python/pyflink/ml/lib/param.py:
##########
@@ -581,3 +581,48 @@ def get_flatten(self) -> bool:
@property
def flatten(self):
return self.get(self.FLATTEN)
+
+
+class HasModelVersionCol(WithParams, ABC):
Review Comment:
Good catch. I have added the test there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on pull request #196: [FLINK-30541] Add Transformer and Estimator for OnlineStandardScaler
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on PR #196:
URL: https://github.com/apache/flink-ml/pull/196#issuecomment-1378314881
> @zhipeng93 Thanks for the PR, left some comments.
Thanks @jiangxin369 for the review. I have addressed your comments in the latest commit. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org