You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/11/11 09:15:52 UTC
[GitHub] [flink-ml] yunfengzhou-hub opened a new pull request #32: [FLINK-24817] Add Naive Bayes implementation
yunfengzhou-hub opened a new pull request #32:
URL: https://github.com/apache/flink-ml/pull/32
## What is the purpose of the change
This PR adds the implementation of Naive Bayes algorithm to Flink ML. This algorithm is implemented in reference to that implemented in Alibaba Alink and Apache Spark, but it uses Flink ML's framework and Flink Datastream.
This PR also adds classes that could convert String to indexed numbers. This function is needed by Naive Bayes so that it only needs to deal with numeric input data.
## Brief change log
This PR adds public classes NaiveBayes and MultiStringIndexer. Users can use NaiveBayes to do training and inference according to the algorithm with the same name. Users can also use MultiStringIndexer to convert strings into indices.
## Verifying this change
The changes are tested by unit tests in MultiStringIndexerTest and NaiveBayesTest.
## Does this pull request potentially affect one of the following parts:
Dependencies (does it add or upgrade a dependency): (yes)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes)
## Documentation
Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? (Java doc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750847767
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
+ return;
+ }
+
+ for (int i = 0; i < featureSize; i++) {
+ Object feature = row.getField(featureColNames[i]);
+ if (feature == null) {
+ continue;
+ }
+ collector.collect(new Tuple4<>(label, i, feature, 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and feature column index.
+ */
+ private static class ValueMapFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Object, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>> {
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> createAccumulator() {
+ return new Tuple4<>(new Object(), -1, new HashMap<>(), 0.);
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> add(
+ Tuple4<Object, Integer, Object, Double> value,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ acc.f3 += value.f3;
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> getResult(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> merge(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc0,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc1) {
+ if (acc0.f0.equals(new Object())) {
+ acc0.f0 = acc1.f0;
+ }
+ if (acc0.f1 == -1) {
+ acc0.f1 = acc1.f1;
+ }
+ for (Map.Entry<Object, Double> entry: acc1.f2.entrySet()) {
Review comment:
OK. I'll make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750846326
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
Review comment:
Yes this PR achieves similar functionality. If this PR is adopted I think we can make Naive Bayes follow this convention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750847113
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
Review comment:
Got it. I have removed the redundant check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r764638257
##########
File path: flink-ml-core/pom.xml
##########
@@ -65,5 +65,12 @@ under the License.
<artifactId>flink-shaded-jackson</artifactId>
<scope>provided</scope>
</dependency>
+
+ <dependency>
+ <groupId>dev.ludovic.netlib</groupId>
+ <artifactId>blas</artifactId>
+ <version>2.2.0</version>
+ <scope>compile</scope>
Review comment:
You are right. I'll remove the scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r764572915
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Integer.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelDataTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelDataStream =
+ NaiveBayesModelData.getModelDataStream(modelDataTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelDataStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(featuresCol, broadcastModelKey));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
Review comment:
Yes, that would be a better choice. I'll make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] gaoyunhaii commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r764945068
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Integer>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Integer>>() {
+ @Override
+ public Tuple2<Vector, Integer> map(Row row) throws Exception {
+ Number number = (Number) row.getField(labelCol);
+ Preconditions.checkArgument(
+ number.intValue() == number.doubleValue());
Review comment:
Perhaps we could give a more meaningful exception instead of `NullPointerExceptioner`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755986990
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Object.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelStream =
+ NaiveBayesModelData.toDataStream(tEnv, modelTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(
+ new PredictLabelFunction(featuresCol, broadcastModelKey)));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.toDataStream(tEnv, modelTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(StreamExecutionEnvironment env, String path)
+ throws IOException {
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ Source<NaiveBayesModelData, ?, ?> source =
+ FileSource.forRecordStreamFormat(
+ new NaiveBayesModelData.ModelDataStreamFormat(),
+ ReadWriteUtils.getDataPaths(path))
+ .build();
+ NaiveBayesModel model = ReadWriteUtils.loadStageParam(path);
+ DataStream<NaiveBayesModelData> modelData =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "modelData");
+ model.setModelData(NaiveBayesModelData.fromDataStream(tEnv, modelData));
+
+ return model;
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ @Override
+ public NaiveBayesModel setModelData(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ modelTable = inputs[0];
+ return this;
+ }
+
+ @Override
+ public Table[] getModelData() {
+ return new Table[] {modelTable};
+ }
+
+ private static class PredictLabelOperator
+ extends AbstractUdfStreamOperator<Row, PredictLabelFunction>
+ implements OneInputStreamOperator<Row, Row> {
+ public PredictLabelOperator(PredictLabelFunction userFunction) {
+ super(userFunction);
+ }
+
+ @Override
+ public void processElement(StreamRecord<Row> streamRecord) {
Review comment:
Actually you can make the change if you use RichMapFunction.
But I think making it a separate function is more clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r757180196
##########
File path: flink-ml-api/src/main/java/org/apache/flink/ml/linalg/BLAS.java
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.linalg;
+
+import org.apache.flink.util.Preconditions;
+
+/** A utility class that provides BLAS routines over matrices and vectors. */
+public class BLAS {
+ /** For level-1 routines, we use Java implementation. */
+ private static final dev.ludovic.netlib.BLAS F2J_BLAS =
+ dev.ludovic.netlib.JavaBLAS.getInstance();
+
+ /** y += a * x . */
+ public static void axpy(double a, double[] x, double[] y) {
Review comment:
In `Vector` we defined a `double[] toArray();` method, so I believe that BLAS can work without direct support for `Vector` subclasses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
lindong28 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r757786000
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Table trainTable;
+ private Table predictTable;
+ private Map<Vector, Integer> expectedOutput;
+ private NaiveBayes estimator;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ Schema trainSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .column("f1", DataTypes.INT())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 0.), 11),
+ Row.of(Vectors.dense(1, 0), 10),
+ Row.of(Vectors.dense(1, 1.), 10))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ trainSchema)
+ .as("features", "label");
+
+ Schema predictSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 1.)),
+ Row.of(Vectors.dense(0, 0.)),
+ Row.of(Vectors.dense(1, 0)),
+ Row.of(Vectors.dense(1, 1.)))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ predictSchema)
+ .as("features");
+
+ expectedOutput =
+ new HashMap<Vector, Integer>() {
+ {
+ put(Vectors.dense(0, 1.), 11);
+ put(Vectors.dense(0, 0.), 11);
+ put(Vectors.dense(1, 0.), 10);
+ put(Vectors.dense(1, 1.), 10);
+ }
+ };
+
+ estimator =
+ new NaiveBayes()
+ .setSmoothing(1.0)
+ .setFeaturesCol("features")
+ .setLabelCol("label")
+ .setPredictionCol("predict")
+ .setModelType("multinomial");
+ }
+
+ /**
+ * Executes a given table and collect its results. Results are returned as a map whose key is
+ * the feature, value is the prediction result.
+ *
+ * @param table A table to be executed and to have its result collected
+ * @param featuresCol Name of the column in the table that contains the features
+ * @param predictionCol Name of the column in the table that contains the prediction result
+ * @return A map containing the collected results
+ */
+ private static Map<Vector, Integer> executeAndCollect(
+ Table table, String featuresCol, String predictionCol) {
+ Map<Vector, Integer> map = new HashMap<>();
+ for (CloseableIterator<Row> it = table.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ map.put(
+ (Vector) row.getField(featuresCol),
+ ((Number) row.getField(predictionCol)).intValue());
+ }
+
+ return map;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testFitAndPredict() throws Exception {
Review comment:
Sound good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750847703
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
Review comment:
Got it. I have added Flink's `Precondition` checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r751924452
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+ private static final String broadcastModelKey = "NaiveBayesModelStream";
Review comment:
`broadcastModelKey` is also used when creating broadcastMap, but it's true that this variable is only used in one public method and its private methods. I'll make `broadcastModelKey` a local variable of `transform()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r748087031
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/common/datastream/EndOfStreamWindows.java
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.triggers.EventTimeTrigger;
+import org.apache.flink.streaming.api.windowing.triggers.Trigger;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+
+import java.util.Collection;
+import java.util.Collections;
+
+/**
+ * A {@link WindowAssigner} that windows elements into windows based on the timestamp of the
+ * elements. Windows cannot overlap.
+ *
+ * <p>For example, in order to window into windows of 1 minute:
+ *
+ * <pre>{@code
+ * DataStream<Tuple2<String, Integer>> in = ...;
+ * KeyedStream<Tuple2<String, Integer>, String> keyed = in.keyBy(...);
+ * WindowedStream<Tuple2<String, Integer>, String, TimeWindow> windowed =
+ * keyed.window(TumblingEventTimeWindows.of(Time.minutes(1)));
+ * }</pre>
+ */
+@PublicEvolving
+public class EndOfStreamWindows extends WindowAssigner<Object, TimeWindow> {
+
+ private static final EndOfStreamWindows INSTANCE = new EndOfStreamWindows();
+
+ private EndOfStreamWindows() {}
+
+ public static EndOfStreamWindows get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Collection<TimeWindow> assignWindows(
+ Object element, long timestamp, WindowAssignerContext context) {
+ return Collections.singletonList(new TimeWindow(Long.MIN_VALUE, Long.MAX_VALUE));
Review comment:
Why do you create a new TimeWindow on each call?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
Review comment:
Shall we use different param classes for `NaiveBayes` and the `NaiveBayesModel`?
It may be confusing to users when call `new NaiveBayesModel().setSmoothing(...)` since `smoothing` is useless during the prediction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r764640893
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Integer>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Integer>>() {
+ @Override
+ public Tuple2<Vector, Integer> map(Row row) throws Exception {
+ Number number = (Number) row.getField(labelCol);
+ Preconditions.checkArgument(
+ number.intValue() == number.doubleValue());
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ number.intValue());
+ }
+ });
+
+ DataStream<NaiveBayesModelData> modelData =
+ input.flatMap(new ExtractFeatureFunction())
+ .keyBy(
+ (KeySelector<Tuple4<Integer, Integer, Double, Double>, Object>)
Review comment:
You are right. I'll remove the type cast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r753077392
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.api.Expressions.$;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private Row[] trainData;
+ private Row[] predictData;
+ private Row[] expectedOutput;
+ private String featuresCol;
+ private String labelCol;
+ private String predictCol;
+ private String modelType;
+ private double smoothing;
+ private boolean isSaveLoad;
+ private String errorMessage;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ schema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.DOUBLE())
+ .column("f1", DataTypes.of(DenseVector.class))
+ .column("f2", DataTypes.DOUBLE())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+
+ predictData = trainData;
+
+ expectedOutput =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0, 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0, 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0, 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0, 10.0)
+ };
+
+ featuresCol = "features";
+ labelCol = "label";
+ predictCol = "predict";
+ modelType = "multinomial";
+ smoothing = 1.0;
+ isSaveLoad = false;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testNaiveBayes() throws Exception {
+ errorMessage = "normal test for Naive Bayes";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testPredictUnseenFeature() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if unseen feature values are met in prediction "
+ + "and the model type is categorical.";
+ predictData =
+ new Row[] {
+ Row.of(1., Vectors.dense(5, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(5, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testVectorWithDiffLen() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if length of feature vectors are not equal.";
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testVectorWithDiffLen2() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if length of feature vectors are not equal.";
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test
+ public void testSaveLoad() throws Exception {
+ errorMessage = "Naive Bayes should be able to save Model to filesystem and load correctly.";
+ isSaveLoad = true;
+ runAndCheck();
+ }
+
+ @Test
+ public void testGetModelData() throws Exception {
+ errorMessage =
+ "Naive Bayes should be able to provide model data with getModelData() method.";
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1.), 11.0), Row.of(2., Vectors.dense(2, 1.), 11.0)
+ };
+ NaiveBayesModel model = getModel();
+ NaiveBayesModelData expected =
+ new NaiveBayesModelData(
+ new HashMap[][] {
+ new HashMap[] {
+ new HashMap<Double, Double>() {
+ {
+ put(2.0, -0.6931471805599453);
+ put(1.0, -0.6931471805599453);
+ }
+ },
+ new HashMap<Double, Double>() {
+ {
+ put(1.0, 0.0);
+ }
+ },
+ },
+ },
+ new double[] {0.0},
+ new double[] {11.0});
+ NaiveBayesModelData actual =
+ NaiveBayesModelData.toDataStream(tEnv, model.getModelData()[0])
+ .executeAndCollect()
+ .next();
+ assertArrayEquals(errorMessage, expected.labels, actual.labels, 1e-5);
+ assertArrayEquals(errorMessage, expected.piArray, actual.piArray, 1e-5);
+ assertArrayEquals(errorMessage, expected.theta[0], actual.theta[0]);
+ }
+
+ @Test
+ public void testSetModelData() throws Exception {
+ errorMessage =
+ "Naive Bayes should be able to provide model data with getModelData() method.";
+
+ NaiveBayesModel modelA = getModel();
+
+ Table modelData = modelA.getModelData()[0];
+ NaiveBayesModel modelB = new NaiveBayesModel().setModelData(modelData);
+ ReadWriteUtils.updateExistingParams(modelB, modelA.getParamMap());
+
+ checkResult(modelB);
+ }
+
+ private void runAndCheck() throws Exception {
+ NaiveBayesModel model = getModel();
+ checkResult(model);
+ }
+
+ private NaiveBayesModel getModel() throws Exception {
+ Table trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(trainData)
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ schema)
+ .as("weight", "features", "label");
+
+ NaiveBayes estimator =
+ new NaiveBayes()
+ .setSmoothing(smoothing)
+ .setFeaturesCol(featuresCol)
+ .setLabelCol(labelCol)
+ .setPredictionCol(predictCol)
+ .setModelType(modelType);
+
+ if (isSaveLoad) {
+ String tempDir = Files.createTempDirectory("").toString();
+ estimator.save(tempDir);
+ env.execute();
+
+ estimator = NaiveBayes.load(tempDir);
+ }
+
+ NaiveBayesModel model = estimator.fit(trainTable);
+
+ if (isSaveLoad) {
+ String tempDir = Files.createTempDirectory("").toString();
+ model.save(tempDir);
+ env.execute();
+
+ model = NaiveBayesModel.load(tempDir);
+ }
+
+ return model;
+ }
+
+ private void checkResult(NaiveBayesModel model) {
+ Table predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(predictData)
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ schema)
+ .as("weight", "features", "label");
+
+ Table output =
+ model.transform(predictTable)[0].select(
+ $("weight"), $("features"), $("label"), $("predict"));
+
+ Object[] actualObjects = IteratorUtils.toArray(output.execute().collect());
+ Row[] actual = new Row[actualObjects.length];
+ for (int i = 0; i < actualObjects.length; i++) {
+ actual[i] = (Row) actualObjects[i];
+ }
+
+ assertEquals(errorMessage, getFrequencyMap(expectedOutput), getFrequencyMap(actual));
+ }
+
+ private static Map<Object, Integer> getFrequencyMap(Row[] rows) {
+ Map<Object, Integer> map = new HashMap<>();
+ for (Row row : rows) {
+ List<Object> list = toList(row);
+ map.put(list, map.getOrDefault(list, 0) + 1);
+ }
+ return map;
+ }
+
+ private static List<Object> toList(Row row) {
Review comment:
classes like this has been replaced by existing tools like `CollectionUtils` in `org.apache.commons`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755766209
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Object.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelStream =
+ NaiveBayesModelData.toDataStream(tEnv, modelTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(
+ new PredictLabelFunction(featuresCol, broadcastModelKey)));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.toDataStream(tEnv, modelTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(StreamExecutionEnvironment env, String path)
+ throws IOException {
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ Source<NaiveBayesModelData, ?, ?> source =
+ FileSource.forRecordStreamFormat(
+ new NaiveBayesModelData.ModelDataStreamFormat(),
+ ReadWriteUtils.getDataPaths(path))
+ .build();
+ NaiveBayesModel model = ReadWriteUtils.loadStageParam(path);
+ DataStream<NaiveBayesModelData> modelData =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "modelData");
+ model.setModelData(NaiveBayesModelData.fromDataStream(tEnv, modelData));
+
+ return model;
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ @Override
+ public NaiveBayesModel setModelData(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ modelTable = inputs[0];
+ return this;
+ }
+
+ @Override
+ public Table[] getModelData() {
+ return new Table[] {modelTable};
+ }
+
+ private static class PredictLabelOperator
+ extends AbstractUdfStreamOperator<Row, PredictLabelFunction>
+ implements OneInputStreamOperator<Row, Row> {
+ public PredictLabelOperator(PredictLabelFunction userFunction) {
+ super(userFunction);
+ }
+
+ @Override
+ public void processElement(StreamRecord<Row> streamRecord) {
Review comment:
I'm afraid that we cannot achieve this by now. The API `withBroadcast` requires that I must pass to it a child class of `AbstractUdfStreamOperator`. A child class of `AbstractStreamOperator` would throw exception at runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
lindong28 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r756974654
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
Review comment:
Do you know what could go wrong if we don't specify `serialVersionUID` for `NaiveBayesModel` and `NaiveBayesModelData`? Spark does not seem to explicitly specify UID for its Estimator/Model subclasses.
If there is no known issue yet, maybe we should remove this UID for simplicity.
If there is known impact, we will need to add this UID for `NaiveBayes`, `KMeans` and `KMeansModel` etc. for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r758054855
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The model data of {@link NaiveBayesModel}.
+ *
+ * <p>This class also provides methods to convert model data between Table and Datastream, and
+ * classes to save/load model data.
+ */
+public class NaiveBayesModelData implements Serializable {
+ public final Map<Double, Double>[][] theta;
+ public final double[] piArray;
+ public final int[] labels;
+
+ public NaiveBayesModelData(Map<Double, Double>[][] theta, double[] piArray, int[] labels) {
+ this.theta = theta;
+ this.piArray = piArray;
+ this.labels = labels;
+ }
+
+ /** Converts the provided modelData Datastream into corresponding Table. */
+ public static Table getModelDataTable(
+ StreamTableEnvironment tEnv, DataStream<NaiveBayesModelData> stream) {
Review comment:
Yes. it makes sense. I'll make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r757201463
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Table trainTable;
+ private Table predictTable;
+ private Map<Vector, Integer> expectedOutput;
+ private NaiveBayes estimator;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ Schema trainSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .column("f1", DataTypes.INT())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 0.), 11),
+ Row.of(Vectors.dense(1, 0), 10),
+ Row.of(Vectors.dense(1, 1.), 10))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ trainSchema)
+ .as("features", "label");
+
+ Schema predictSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 1.)),
+ Row.of(Vectors.dense(0, 0.)),
+ Row.of(Vectors.dense(1, 0)),
+ Row.of(Vectors.dense(1, 1.)))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ predictSchema)
+ .as("features");
+
+ expectedOutput =
+ new HashMap<Vector, Integer>() {
+ {
+ put(Vectors.dense(0, 1.), 11);
+ put(Vectors.dense(0, 0.), 11);
+ put(Vectors.dense(1, 0.), 10);
+ put(Vectors.dense(1, 1.), 10);
+ }
+ };
+
+ estimator =
+ new NaiveBayes()
+ .setSmoothing(1.0)
+ .setFeaturesCol("features")
+ .setLabelCol("label")
+ .setPredictionCol("predict")
+ .setModelType("multinomial");
+ }
+
+ /**
+ * Executes a given table and collect its results. Results are returned as a map whose key is
+ * the feature, value is the prediction result.
+ *
+ * @param table A table to be executed and to have its result collected
+ * @param featuresCol Name of the column in the table that contains the features
+ * @param predictionCol Name of the column in the table that contains the prediction result
+ * @return A map containing the collected results
+ */
+ private static Map<Vector, Integer> executeAndCollect(
+ Table table, String featuresCol, String predictionCol) {
+ Map<Vector, Integer> map = new HashMap<>();
+ for (CloseableIterator<Row> it = table.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ map.put(
+ (Vector) row.getField(featuresCol),
+ ((Number) row.getField(predictionCol)).intValue());
+ }
+
+ return map;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testFitAndPredict() throws Exception {
+ NaiveBayesModel model = estimator.fit(trainTable);
+ Table outputTable = model.transform(predictTable)[0];
+ Map<Vector, Integer> actualOutput =
+ executeAndCollect(outputTable, model.getFeaturesCol(), model.getPredictionCol());
+ assertEquals(expectedOutput, actualOutput);
+ }
+
+ @Test(expected = Exception.class)
+ public void testPredictUnseenFeature() throws Exception {
Review comment:
I think `testPredictUnseenFeature` should fail naturally while the program cannot find corresponding key of the feature. Adding a check and set detailed error message might have a bit influence on the performance of this algorithm. Other tests that expect an exception do need to check the message and I'll make this change on those tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
lindong28 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r757785775
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Table trainTable;
+ private Table predictTable;
+ private Map<Vector, Integer> expectedOutput;
+ private NaiveBayes estimator;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ Schema trainSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .column("f1", DataTypes.INT())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 0.), 11),
+ Row.of(Vectors.dense(1, 0), 10),
+ Row.of(Vectors.dense(1, 1.), 10))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ trainSchema)
+ .as("features", "label");
+
+ Schema predictSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 1.)),
+ Row.of(Vectors.dense(0, 0.)),
+ Row.of(Vectors.dense(1, 0)),
+ Row.of(Vectors.dense(1, 1.)))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ predictSchema)
+ .as("features");
+
+ expectedOutput =
+ new HashMap<Vector, Integer>() {
+ {
+ put(Vectors.dense(0, 1.), 11);
+ put(Vectors.dense(0, 0.), 11);
+ put(Vectors.dense(1, 0.), 10);
+ put(Vectors.dense(1, 1.), 10);
+ }
+ };
+
+ estimator =
+ new NaiveBayes()
+ .setSmoothing(1.0)
+ .setFeaturesCol("features")
+ .setLabelCol("label")
+ .setPredictionCol("predict")
+ .setModelType("multinomial");
+ }
+
+ /**
+ * Executes a given table and collect its results. Results are returned as a map whose key is
+ * the feature, value is the prediction result.
+ *
+ * @param table A table to be executed and to have its result collected
+ * @param featuresCol Name of the column in the table that contains the features
+ * @param predictionCol Name of the column in the table that contains the prediction result
+ * @return A map containing the collected results
+ */
+ private static Map<Vector, Integer> executeAndCollect(
+ Table table, String featuresCol, String predictionCol) {
+ Map<Vector, Integer> map = new HashMap<>();
+ for (CloseableIterator<Row> it = table.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ map.put(
+ (Vector) row.getField(featuresCol),
+ ((Number) row.getField(predictionCol)).intValue());
+ }
+
+ return map;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testFitAndPredict() throws Exception {
+ NaiveBayesModel model = estimator.fit(trainTable);
+ Table outputTable = model.transform(predictTable)[0];
+ Map<Vector, Integer> actualOutput =
+ executeAndCollect(outputTable, model.getFeaturesCol(), model.getPredictionCol());
+ assertEquals(expectedOutput, actualOutput);
+ }
+
+ @Test(expected = Exception.class)
+ public void testPredictUnseenFeature() throws Exception {
Review comment:
Sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r757201463
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Table trainTable;
+ private Table predictTable;
+ private Map<Vector, Integer> expectedOutput;
+ private NaiveBayes estimator;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ Schema trainSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .column("f1", DataTypes.INT())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 0.), 11),
+ Row.of(Vectors.dense(1, 0), 10),
+ Row.of(Vectors.dense(1, 1.), 10))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ trainSchema)
+ .as("features", "label");
+
+ Schema predictSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 1.)),
+ Row.of(Vectors.dense(0, 0.)),
+ Row.of(Vectors.dense(1, 0)),
+ Row.of(Vectors.dense(1, 1.)))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ predictSchema)
+ .as("features");
+
+ expectedOutput =
+ new HashMap<Vector, Integer>() {
+ {
+ put(Vectors.dense(0, 1.), 11);
+ put(Vectors.dense(0, 0.), 11);
+ put(Vectors.dense(1, 0.), 10);
+ put(Vectors.dense(1, 1.), 10);
+ }
+ };
+
+ estimator =
+ new NaiveBayes()
+ .setSmoothing(1.0)
+ .setFeaturesCol("features")
+ .setLabelCol("label")
+ .setPredictionCol("predict")
+ .setModelType("multinomial");
+ }
+
+ /**
+ * Executes a given table and collect its results. Results are returned as a map whose key is
+ * the feature, value is the prediction result.
+ *
+ * @param table A table to be executed and to have its result collected
+ * @param featuresCol Name of the column in the table that contains the features
+ * @param predictionCol Name of the column in the table that contains the prediction result
+ * @return A map containing the collected results
+ */
+ private static Map<Vector, Integer> executeAndCollect(
+ Table table, String featuresCol, String predictionCol) {
+ Map<Vector, Integer> map = new HashMap<>();
+ for (CloseableIterator<Row> it = table.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ map.put(
+ (Vector) row.getField(featuresCol),
+ ((Number) row.getField(predictionCol)).intValue());
+ }
+
+ return map;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testFitAndPredict() throws Exception {
+ NaiveBayesModel model = estimator.fit(trainTable);
+ Table outputTable = model.transform(predictTable)[0];
+ Map<Vector, Integer> actualOutput =
+ executeAndCollect(outputTable, model.getFeaturesCol(), model.getPredictionCol());
+ assertEquals(expectedOutput, actualOutput);
+ }
+
+ @Test(expected = Exception.class)
+ public void testPredictUnseenFeature() throws Exception {
Review comment:
I think `testPredictUnseenFeature` should fail naturally while the program cannot find corresponding value of the feature. Adding a check and set detailed error message might have a bit influence on the performance of this algorithm. Other tests that expect an exception do need to check the message and I'll make this change on those tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r757211597
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Table trainTable;
+ private Table predictTable;
+ private Map<Vector, Integer> expectedOutput;
+ private NaiveBayes estimator;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ Schema trainSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .column("f1", DataTypes.INT())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 0.), 11),
+ Row.of(Vectors.dense(1, 0), 10),
+ Row.of(Vectors.dense(1, 1.), 10))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ trainSchema)
+ .as("features", "label");
+
+ Schema predictSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 1.)),
+ Row.of(Vectors.dense(0, 0.)),
+ Row.of(Vectors.dense(1, 0)),
+ Row.of(Vectors.dense(1, 1.)))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ predictSchema)
+ .as("features");
+
+ expectedOutput =
+ new HashMap<Vector, Integer>() {
+ {
+ put(Vectors.dense(0, 1.), 11);
+ put(Vectors.dense(0, 0.), 11);
+ put(Vectors.dense(1, 0.), 10);
+ put(Vectors.dense(1, 1.), 10);
+ }
+ };
+
+ estimator =
+ new NaiveBayes()
+ .setSmoothing(1.0)
+ .setFeaturesCol("features")
+ .setLabelCol("label")
+ .setPredictionCol("predict")
+ .setModelType("multinomial");
+ }
+
+ /**
+ * Executes a given table and collect its results. Results are returned as a map whose key is
+ * the feature, value is the prediction result.
+ *
+ * @param table A table to be executed and to have its result collected
+ * @param featuresCol Name of the column in the table that contains the features
+ * @param predictionCol Name of the column in the table that contains the prediction result
+ * @return A map containing the collected results
+ */
+ private static Map<Vector, Integer> executeAndCollect(
+ Table table, String featuresCol, String predictionCol) {
+ Map<Vector, Integer> map = new HashMap<>();
+ for (CloseableIterator<Row> it = table.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ map.put(
+ (Vector) row.getField(featuresCol),
+ ((Number) row.getField(predictionCol)).intValue());
+ }
+
+ return map;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testFitAndPredict() throws Exception {
Review comment:
OK. I'll make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755671222
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/clustering/KMeansTest.java
##########
@@ -83,43 +97,19 @@ public void before() {
dataTable = tEnv.fromDataStream(env.fromCollection(DATA), schema).as("features");
}
- // Executes the graph and returns a map which maps points to clusterId.
- private static Map<DenseVector, Integer> executeAndCollect(
- Table output, String featureCol, String predictionCol) throws Exception {
- StreamTableEnvironment tEnv =
- (StreamTableEnvironment) ((TableImpl) output).getTableEnvironment();
-
- DataStream<Tuple2<DenseVector, Integer>> stream =
- tEnv.toDataStream(output)
- .map(
- new MapFunction<Row, Tuple2<DenseVector, Integer>>() {
- @Override
- public Tuple2<DenseVector, Integer> map(Row row) {
- return Tuple2.of(
- (DenseVector) row.getField(featureCol),
- (Integer) row.getField(predictionCol));
- }
- });
-
- List<Tuple2<DenseVector, Integer>> pointsWithClusterId =
- IteratorUtils.toList(stream.executeAndCollect());
-
- Map<DenseVector, Integer> clusterIdByPoints = new HashMap<>();
- for (Tuple2<DenseVector, Integer> entry : pointsWithClusterId) {
- clusterIdByPoints.put(entry.f0, entry.f1);
- }
- return clusterIdByPoints;
- }
-
- private static void verifyClusteringResult(
- Map<DenseVector, Integer> clusterIdByPoints, List<List<Integer>> groups) {
- for (List<Integer> group : groups) {
- for (int i = 1; i < group.size(); i++) {
- assertEquals(
- clusterIdByPoints.get(DATA.get(group.get(0))),
- clusterIdByPoints.get(DATA.get(group.get(i))));
+ private static List<Set<DenseVector>> executeAndCollect(
Review comment:
OK. I'll fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750250851
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
Review comment:
Is this helpful in this case?
https://github.com/apache/flink-ml/pull/30
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r764666357
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Integer.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelDataTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelDataStream =
+ NaiveBayesModelData.getModelDataStream(modelDataTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelDataStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(featuresCol, broadcastModelKey));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
Review comment:
I agree. The implementation in LogisticRegression cannot exactly meet the needs for NaiveBayes and Kmeans, so I made some adjustment and applied this practice to these two algorithms.
Still, the current implementation cannot satisfy all possible needs of modelData, which could be more than one. We may need further discussion about how to make these utility methods cover more cases like this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r749958347
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesParams.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.ml.param.DoubleParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.shared.colname.HasFeatureCols;
+import org.apache.flink.ml.param.shared.colname.HasLabelCol;
+import org.apache.flink.ml.param.shared.colname.HasPredictionCol;
+
+/**
+ * Parameters of naive bayes training process.
+ */
+public interface NaiveBayesParams<T> extends
Review comment:
OK. I'll add this param to NaiveBayes.
One consideration is that while spark supports setting the type of a whole Naive Bayes model, Alink allows finer granularity to define the model type for each feature column. I agree that we just need to define the type of a whole model in the foreseeable future, but if users need to define type for each column they will need to create multiple NaiveBayesModel and use some bagging algorithm to unify them into one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750847799
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
+ return;
+ }
+
+ for (int i = 0; i < featureSize; i++) {
+ Object feature = row.getField(featureColNames[i]);
+ if (feature == null) {
+ continue;
+ }
+ collector.collect(new Tuple4<>(label, i, feature, 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and feature column index.
+ */
+ private static class ValueMapFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Object, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>> {
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> createAccumulator() {
+ return new Tuple4<>(new Object(), -1, new HashMap<>(), 0.);
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> add(
+ Tuple4<Object, Integer, Object, Double> value,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ acc.f3 += value.f3;
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> getResult(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> merge(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc0,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc1) {
+ if (acc0.f0.equals(new Object())) {
Review comment:
Yes, sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750845531
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.ObjectArrayTypeInfo;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.Map;
+
+/**
+ * The model data of {@link NaiveBayesModel}.
+ */
+public class NaiveBayesModelData implements Serializable {
+ private static final long serialVersionUID = 3919917903722286395L;
+ public final String[] featureNames;
+ public final Map<Object, Double>[][] theta;
+ public final double[] piArray;
+ public final Object[] label;
+
+ public static final Schema SCHEMA =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(NaiveBayesModelData.class))
+ .build();
+
+ public NaiveBayesModelData(String[] featureNames, Map<Object, Double>[][] theta, double[] piArray, Object[] label) {
+ this.featureNames = featureNames;
+ this.theta = theta;
+ this.piArray = piArray;
+ this.label = label;
+ }
+
+ /** Encoder for the KMeans model data. */
Review comment:
OK. I have fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
lindong28 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750878843
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasFeatureSize.java
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.IntParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.WithParams;
+
+/** Interface for the shared featureLength param. */
+public interface HasFeatureSize<T> extends WithParams<T> {
Review comment:
Can we remove this param and derive this information from the input data?
##########
File path: flink-ml-api/pom.xml
##########
@@ -65,5 +65,19 @@ under the License.
<artifactId>flink-shaded-jackson</artifactId>
<scope>provided</scope>
</dependency>
+
+ <dependency>
+ <groupId>com.github.fommil.netlib</groupId>
Review comment:
Do we still need `com.github.fommil.netlib`?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelParams.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.ml.common.param.HasFeatureSize;
+import org.apache.flink.ml.common.param.HasFeaturesCol;
+import org.apache.flink.ml.common.param.HasPredictionCol;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.StringParam;
+
+/** Parameters of naive bayes training process. */
Review comment:
Maybe avoid introducing concepts such as `process` here and follow the same comments patterns as the existing algorithm?
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.api.Expressions.$;
+
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private Row[] trainData;
+ private Row[] predictData;
+ private Row[] expectedOutput;
+ private String featuresCol;
+ private int featureSize;
+ private String labelCol;
+ private String predictCol;
+ private String modelType;
+ private double smoothing;
+ private boolean isSaveLoad;
+ private String errorMessage;
+
+ @Before
+ public void Setup() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ schema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.DOUBLE())
+ .column("f1", DataTypes.of(DenseVector.class))
+ .column("f2", DataTypes.DOUBLE())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+
+ predictData = trainData;
+
+ expectedOutput =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0, 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0, 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0, 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0, 10.0)
+ };
+
+ featuresCol = "features";
+ featureSize = 5;
+ labelCol = "label";
+ predictCol = "predict";
+ modelType = "categorical";
+ smoothing = 1.0;
+ isSaveLoad = false;
+ }
+
+ @Test
+ public void testNaiveBayes() throws Exception {
+ errorMessage = "normal test for Naive Bayes";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testInputNotContainFeature() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if some feature columns are missing from train data";
+ featuresCol = "feature2";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testPredictUnseenFeature() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if unseen feature values are met in prediction "
+ + "and the model type is categorical.";
+ predictData =
+ new Row[] {
+ Row.of(1., Vectors.dense(5, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(5, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testEmptyLabel() throws Exception {
Review comment:
It seems a bit verbose to have a dedicated test for each param. We probably don't want to do this for every parameter of every Estimator/Transformer.
And the purpose of this test seems to have already been covered by `StageTest::testValidators`. We can add unit tests there if needed.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasLabelCol.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.StringParam;
+import org.apache.flink.ml.param.WithParams;
+
+/**
+ * Param of the name of the label column in the input table.
+ *
+ * @param <T>
+ */
+public interface HasLabelCol<T> extends WithParams<T> {
+ Param<String> LABEL_COL =
+ new StringParam(
+ "labelCol",
+ "Name of the label column in the input table.",
+ null,
Review comment:
How about changing the doc to `Label column name.` to stay consistent with `HasFeaturesCol`?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelParams.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.ml.common.param.HasFeatureSize;
+import org.apache.flink.ml.common.param.HasFeaturesCol;
+import org.apache.flink.ml.common.param.HasPredictionCol;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.StringParam;
+
+/** Parameters of naive bayes training process. */
+public interface NaiveBayesModelParams<T>
+ extends HasFeaturesCol<T>, HasFeatureSize<T>, HasPredictionCol<T> {
+ Param<String> MODEL_TYPE =
+ new StringParam(
+ "modelType",
+ "Type of the Naive Bayes model.",
Review comment:
It may be useful to specify the supported options here, as is done in `HasDistanceMeasure` and in Spark.
For example, `The model type. Supported options: 'categorical'.`
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.api.Expressions.$;
+
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private Row[] trainData;
+ private Row[] predictData;
+ private Row[] expectedOutput;
+ private String featuresCol;
+ private int featureSize;
+ private String labelCol;
+ private String predictCol;
+ private String modelType;
+ private double smoothing;
+ private boolean isSaveLoad;
+ private String errorMessage;
+
+ @Before
+ public void Setup() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ schema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.DOUBLE())
+ .column("f1", DataTypes.of(DenseVector.class))
+ .column("f2", DataTypes.DOUBLE())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+
+ predictData = trainData;
+
+ expectedOutput =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0, 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0, 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0, 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0, 10.0)
+ };
+
+ featuresCol = "features";
+ featureSize = 5;
+ labelCol = "label";
+ predictCol = "predict";
+ modelType = "categorical";
+ smoothing = 1.0;
+ isSaveLoad = false;
+ }
+
+ @Test
+ public void testNaiveBayes() throws Exception {
+ errorMessage = "normal test for Naive Bayes";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testInputNotContainFeature() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if some feature columns are missing from train data";
+ featuresCol = "feature2";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testPredictUnseenFeature() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if unseen feature values are met in prediction "
+ + "and the model type is categorical.";
+ predictData =
+ new Row[] {
+ Row.of(1., Vectors.dense(5, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(5, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testEmptyLabel() throws Exception {
Review comment:
It seems a bit verbose to have a dedicated test for each param. We probably don't want to do this for every parameter of every Estimator/Transformer.
And the purpose of this test seems to have already been covered by `StageTest::testValidators`. We can add unit tests there if needed.
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.api.Expressions.$;
+
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private Row[] trainData;
+ private Row[] predictData;
+ private Row[] expectedOutput;
+ private String featuresCol;
+ private int featureSize;
+ private String labelCol;
+ private String predictCol;
+ private String modelType;
+ private double smoothing;
+ private boolean isSaveLoad;
+ private String errorMessage;
+
+ @Before
+ public void Setup() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ schema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.DOUBLE())
+ .column("f1", DataTypes.of(DenseVector.class))
+ .column("f2", DataTypes.DOUBLE())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+
+ predictData = trainData;
+
+ expectedOutput =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0, 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0, 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0, 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0, 10.0)
+ };
+
+ featuresCol = "features";
+ featureSize = 5;
+ labelCol = "label";
+ predictCol = "predict";
+ modelType = "categorical";
+ smoothing = 1.0;
+ isSaveLoad = false;
+ }
+
+ @Test
+ public void testNaiveBayes() throws Exception {
+ errorMessage = "normal test for Naive Bayes";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testInputNotContainFeature() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if some feature columns are missing from train data";
+ featuresCol = "feature2";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testPredictUnseenFeature() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if unseen feature values are met in prediction "
+ + "and the model type is categorical.";
+ predictData =
+ new Row[] {
+ Row.of(1., Vectors.dense(5, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(5, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testEmptyLabel() throws Exception {
+ errorMessage = "Naive Bayes should throw exception if label is empty string";
+ labelCol = "";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testNullLabel() throws Exception {
+ errorMessage = "Naive Bayes should throw exception if label is null";
+ labelCol = null;
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testNullLabelValue() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception of train or predict data contains null label.";
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), null),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), null),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), null),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testInputNotContainLabel() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if input table schema does not contain label column.";
+ labelCol = "non-label";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testNullPredict() throws Exception {
+ errorMessage = "Naive Bayes should throw exception if predict col is not set.";
+ predictCol = null;
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testNullModelType() throws Exception {
+ errorMessage = "Naive Bayes should throw exception if model type is not set.";
+ modelType = null;
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testVectorWithDiffLen() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if length of feature vectors are not equal.";
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test
+ public void testSaveLoad() throws Exception {
+ errorMessage = "Naive Bayes should be able to save Model to filesystem and load correctly.";
+ isSaveLoad = true;
+ runAndCheck();
+ }
+
+ private void runAndCheck() throws Exception {
+ Table trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(trainData)
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ schema)
+ .as("weight", "features", "label");
+ Table predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(predictData)
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ schema)
+ .as("weight", "features", "label");
+
+ NaiveBayes estimator = new NaiveBayes();
+ estimator.setSmoothing(smoothing);
+ if (featuresCol != null) estimator.setFeaturesCol(featuresCol);
Review comment:
Can we just remove the `if` and set the param directly?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/** Naive Bayes Predictor. */
Review comment:
Can you rephrase the comments here following KMeans or Spark's convention? It is preferred not to introduce `Predictor` as a new concept here.
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.api.Expressions.$;
+
+public class NaiveBayesTest {
Review comment:
Can we add test to cover `setModelData`, `getModelData` and default param values? Maybe use KMeansTest for example.
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.api.Expressions.$;
+
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private Row[] trainData;
+ private Row[] predictData;
+ private Row[] expectedOutput;
+ private String featuresCol;
+ private int featureSize;
+ private String labelCol;
+ private String predictCol;
+ private String modelType;
+ private double smoothing;
+ private boolean isSaveLoad;
+ private String errorMessage;
+
+ @Before
+ public void Setup() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ schema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.DOUBLE())
+ .column("f1", DataTypes.of(DenseVector.class))
+ .column("f2", DataTypes.DOUBLE())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+
+ predictData = trainData;
+
+ expectedOutput =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0, 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0, 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0, 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0, 10.0)
+ };
+
+ featuresCol = "features";
+ featureSize = 5;
+ labelCol = "label";
+ predictCol = "predict";
+ modelType = "categorical";
+ smoothing = 1.0;
+ isSaveLoad = false;
+ }
+
+ @Test
+ public void testNaiveBayes() throws Exception {
+ errorMessage = "normal test for Naive Bayes";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testInputNotContainFeature() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if some feature columns are missing from train data";
+ featuresCol = "feature2";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testPredictUnseenFeature() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if unseen feature values are met in prediction "
+ + "and the model type is categorical.";
+ predictData =
+ new Row[] {
+ Row.of(1., Vectors.dense(5, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(5, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testEmptyLabel() throws Exception {
+ errorMessage = "Naive Bayes should throw exception if label is empty string";
+ labelCol = "";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testNullLabel() throws Exception {
+ errorMessage = "Naive Bayes should throw exception if label is null";
+ labelCol = null;
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testNullLabelValue() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception of train or predict data contains null label.";
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), null),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), null),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), null),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testInputNotContainLabel() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if input table schema does not contain label column.";
+ labelCol = "non-label";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testNullPredict() throws Exception {
+ errorMessage = "Naive Bayes should throw exception if predict col is not set.";
+ predictCol = null;
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testNullModelType() throws Exception {
+ errorMessage = "Naive Bayes should throw exception if model type is not set.";
+ modelType = null;
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testVectorWithDiffLen() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if length of feature vectors are not equal.";
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test
+ public void testSaveLoad() throws Exception {
+ errorMessage = "Naive Bayes should be able to save Model to filesystem and load correctly.";
+ isSaveLoad = true;
+ runAndCheck();
+ }
+
+ private void runAndCheck() throws Exception {
+ Table trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(trainData)
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ schema)
+ .as("weight", "features", "label");
+ Table predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(predictData)
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ schema)
+ .as("weight", "features", "label");
+
+ NaiveBayes estimator = new NaiveBayes();
Review comment:
It is preferred to follow best practice when writing test code, since it is what users will read and probably copy.
We can use `NaiveBayes estimator = new NaiveBayes().setSmoothing(smoothing)` here.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,338 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using Bayes theorem based
+ * on independent assumption. It is an independent feature model. The input feature can be continual
+ * or categorical.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ final String featuresCol = getFeaturesCol();
+ final int featureSize = getFeatureSize();
+ final String labelCol = getLabelCol();
+ final String predictionCol = getPredictionCol();
Review comment:
Should we remove `predictionCol` since it is not used?
Same for other variables that are not used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
lindong28 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750879940
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelParams.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.ml.common.param.HasFeatureSize;
+import org.apache.flink.ml.common.param.HasFeaturesCol;
+import org.apache.flink.ml.common.param.HasPredictionCol;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.StringParam;
+
+/** Parameters of naive bayes training process. */
+public interface NaiveBayesModelParams<T>
+ extends HasFeaturesCol<T>, HasFeatureSize<T>, HasPredictionCol<T> {
+ Param<String> MODEL_TYPE =
+ new StringParam(
+ "modelType",
+ "Type of the Naive Bayes model.",
Review comment:
It may be useful to specify the supported options here, as is done in `HasDistanceMeasure` and in Spark.
For example, `The model type. Supported options: 'categorical'.`
And Spark does not have `categorical` as model type. I think it is preferred to follow Spark's convention if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r749921048
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/common/datastream/EndOfStreamWindows.java
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.triggers.EventTimeTrigger;
+import org.apache.flink.streaming.api.windowing.triggers.Trigger;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+
+import java.util.Collection;
+import java.util.Collections;
+
+/**
+ * A {@link WindowAssigner} that windows elements into windows based on the timestamp of the
+ * elements. Windows cannot overlap.
+ *
+ * <p>For example, in order to window into windows of 1 minute:
+ *
+ * <pre>{@code
+ * DataStream<Tuple2<String, Integer>> in = ...;
+ * KeyedStream<Tuple2<String, Integer>, String> keyed = in.keyBy(...);
+ * WindowedStream<Tuple2<String, Integer>, String, TimeWindow> windowed =
+ * keyed.window(TumblingEventTimeWindows.of(Time.minutes(1)));
+ * }</pre>
+ */
+@PublicEvolving
+public class EndOfStreamWindows extends WindowAssigner<Object, TimeWindow> {
+
+ private static final EndOfStreamWindows INSTANCE = new EndOfStreamWindows();
+
+ private EndOfStreamWindows() {}
+
+ public static EndOfStreamWindows get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Collection<TimeWindow> assignWindows(
+ Object element, long timestamp, WindowAssignerContext context) {
+ return Collections.singletonList(new TimeWindow(Long.MIN_VALUE, Long.MAX_VALUE));
Review comment:
I just confirmed with Lin Dong as this is his code. He agrees that we can create a global static variable here and reuse it every time we call `assignWindows()`. He will make this change in next version of his code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
lindong28 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755192622
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/clustering/KMeansTest.java
##########
@@ -158,10 +148,11 @@ public void testFeaturePredictionParam() throws Exception {
assertEquals(
Arrays.asList("test_feature", "test_prediction"),
output.getResolvedSchema().getColumnNames());
- Map<DenseVector, Integer> clusterIdByPoints =
- executeAndCollect(output, kmeans.getFeaturesCol(), kmeans.getPredictionCol());
- verifyClusteringResult(
- clusterIdByPoints, Arrays.asList(Arrays.asList(0, 1, 2), Arrays.asList(3, 4, 5)));
+ assertTrue(
+ CollectionUtils.isEqualCollection(
+ executeAndCollect(
Review comment:
nits: this statement seems a bit too long. Maybe move `executeAndCollect` to a separate statement?
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/clustering/KMeansTest.java
##########
@@ -83,43 +97,19 @@ public void before() {
dataTable = tEnv.fromDataStream(env.fromCollection(DATA), schema).as("features");
}
- // Executes the graph and returns a map which maps points to clusterId.
- private static Map<DenseVector, Integer> executeAndCollect(
- Table output, String featureCol, String predictionCol) throws Exception {
- StreamTableEnvironment tEnv =
- (StreamTableEnvironment) ((TableImpl) output).getTableEnvironment();
-
- DataStream<Tuple2<DenseVector, Integer>> stream =
- tEnv.toDataStream(output)
- .map(
- new MapFunction<Row, Tuple2<DenseVector, Integer>>() {
- @Override
- public Tuple2<DenseVector, Integer> map(Row row) {
- return Tuple2.of(
- (DenseVector) row.getField(featureCol),
- (Integer) row.getField(predictionCol));
- }
- });
-
- List<Tuple2<DenseVector, Integer>> pointsWithClusterId =
- IteratorUtils.toList(stream.executeAndCollect());
-
- Map<DenseVector, Integer> clusterIdByPoints = new HashMap<>();
- for (Tuple2<DenseVector, Integer> entry : pointsWithClusterId) {
- clusterIdByPoints.put(entry.f0, entry.f1);
- }
- return clusterIdByPoints;
- }
-
- private static void verifyClusteringResult(
- Map<DenseVector, Integer> clusterIdByPoints, List<List<Integer>> groups) {
- for (List<Integer> group : groups) {
- for (int i = 1; i < group.size(); i++) {
- assertEquals(
- clusterIdByPoints.get(DATA.get(group.get(0))),
- clusterIdByPoints.get(DATA.get(group.get(i))));
+ private static List<Set<DenseVector>> executeAndCollect(
Review comment:
nits: Can we add comments for this method?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.Map;
+
+/** The model data of {@link NaiveBayesModel}. Also provides classes to save/load model data. */
+public class NaiveBayesModelData implements Serializable {
+ private static final long serialVersionUID = 3919917903722286395L;
+ public final Map<Double, Double>[][] theta;
+ public final double[] piArray;
+ public final double[] labels;
+
+ // Empty constructor is used when Kyro deserializes loaded model data.
+ public NaiveBayesModelData() {
+ this(null, null, null);
+ }
+
+ public NaiveBayesModelData(Map<Double, Double>[][] theta, double[] piArray, double[] labels) {
+ this.theta = theta;
+ this.piArray = piArray;
+ this.labels = labels;
+ }
+
+ public static Table fromDataStream(
Review comment:
Can we have Java doc here since this is a public method?
Should we also update KMeans to use the same pattern?
Can we use more informational names here, e.g. `getModelDataTable` and `getModelDataStream`?
Same for `toDataStream`.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Double>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Double>>() {
+ @Override
+ public Tuple2<Vector, Double> map(Row row) throws Exception {
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ (Double) row.getField(labelCol));
+ }
+ });
+
+ DataStream<NaiveBayesModelData> naiveBayesModel =
+ input.flatMap(new FlattenFunction())
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce(
+ (ReduceFunction<Tuple4<Double, Integer, Double, Double>>)
+ (t0, t1) -> {
+ t0.f3 += t1.f3;
+ return t0;
+ })
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy(
+ (KeySelector<
+ Tuple4<
+ Double,
+ Integer,
+ Map<Double, Double>,
+ Double>,
+ Object>)
+ value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction())
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(smoothing));
+
+ NaiveBayesModel model =
+ new NaiveBayesModel()
+ .setModelData(NaiveBayesModelData.fromDataStream(tEnv, naiveBayesModel));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(StreamExecutionEnvironment env, String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value,
+ * weight.
+ */
+ private static class FlattenFunction
+ implements FlatMapFunction<
+ Tuple2<Vector, Double>, Tuple4<Double, Integer, Double, Double>> {
+ @Override
+ public void flatMap(
+ Tuple2<Vector, Double> value,
+ Collector<Tuple4<Double, Integer, Double, Double>> collector) {
+ Preconditions.checkNotNull(value.f1);
+ for (int i = 0; i < value.f0.size(); i++) {
+ collector.collect(new Tuple4<>(value.f1, i, value.f0.get(i), 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and
+ * feature column index.
+ */
+ private static class ValueMapFunction
+ implements AggregateFunction<
+ Tuple4<Double, Integer, Double, Double>,
+ Tuple3<Double, Integer, Map<Double, Double>>,
+ Tuple4<Double, Integer, Map<Double, Double>, Double>> {
+
+ @Override
+ public Tuple3<Double, Integer, Map<Double, Double>> createAccumulator() {
+ return new Tuple3<>(0., -1, new HashMap<>());
+ }
+
+ @Override
+ public Tuple3<Double, Integer, Map<Double, Double>> add(
+ Tuple4<Double, Integer, Double, Double> value,
+ Tuple3<Double, Integer, Map<Double, Double>> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Double, Integer, Map<Double, Double>, Double> getResult(
+ Tuple3<Double, Integer, Map<Double, Double>> acc) {
+ double weightSum = acc.f2.values().stream().mapToDouble(f -> f).sum();
+ return new Tuple4<>(acc.f0, acc.f1, acc.f2, weightSum);
+ }
+
+ @Override
+ public Tuple3<Double, Integer, Map<Double, Double>> merge(
+ Tuple3<Double, Integer, Map<Double, Double>> acc0,
+ Tuple3<Double, Integer, Map<Double, Double>> acc1) {
+ Preconditions.checkArgument(acc0.f1 != -1);
+ acc0.f2.putAll(acc1.f2);
+ return acc0;
+ }
+ }
+
+ /** Function to aggregate maps under the same label into arrays. array len = featureSize. */
+ private static class MapArrayFunction
Review comment:
It is not very easy to understand the purpose of this function.
Could you add comments explaining the meaning of fields of input and accumulators?
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/clustering/KMeansTest.java
##########
@@ -67,6 +69,18 @@
Vectors.dense(9.6, 0.0));
private StreamExecutionEnvironment env;
private StreamTableEnvironment tEnv;
+ private static List<Set<DenseVector>> expectedGroup =
Review comment:
nits: expectedGroup -> expectedGroups
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+
+import static org.apache.flink.table.api.Expressions.$;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private List<Row> trainData;
+ private List<Row> predictData;
+ private List<Row> expectedOutput;
+ private boolean isSaveLoad;
Review comment:
I believe the general rules of thumb is to make variable a member variable if this variable is written only in `before`. For variables written in individual tests (e.g. `isSaveLoad`), we typically pass this variable directly from the test method to the callee method.
I could be wrong. Please feel free to verify this practice in the Flink codebase. If this is indeed the case for most tests, can we follow this practice here?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Double>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Double>>() {
+ @Override
+ public Tuple2<Vector, Double> map(Row row) throws Exception {
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ (Double) row.getField(labelCol));
+ }
+ });
+
+ DataStream<NaiveBayesModelData> naiveBayesModel =
+ input.flatMap(new FlattenFunction())
Review comment:
Can we rename `FlattenFunction` to something more meaningful?
Same for `ValueMapFunction` and `MapArrayFunction`.
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/clustering/KMeansTest.java
##########
@@ -83,43 +97,19 @@ public void before() {
dataTable = tEnv.fromDataStream(env.fromCollection(DATA), schema).as("features");
}
- // Executes the graph and returns a map which maps points to clusterId.
- private static Map<DenseVector, Integer> executeAndCollect(
- Table output, String featureCol, String predictionCol) throws Exception {
- StreamTableEnvironment tEnv =
- (StreamTableEnvironment) ((TableImpl) output).getTableEnvironment();
-
- DataStream<Tuple2<DenseVector, Integer>> stream =
- tEnv.toDataStream(output)
- .map(
- new MapFunction<Row, Tuple2<DenseVector, Integer>>() {
- @Override
- public Tuple2<DenseVector, Integer> map(Row row) {
- return Tuple2.of(
- (DenseVector) row.getField(featureCol),
- (Integer) row.getField(predictionCol));
- }
- });
-
- List<Tuple2<DenseVector, Integer>> pointsWithClusterId =
- IteratorUtils.toList(stream.executeAndCollect());
-
- Map<DenseVector, Integer> clusterIdByPoints = new HashMap<>();
- for (Tuple2<DenseVector, Integer> entry : pointsWithClusterId) {
- clusterIdByPoints.put(entry.f0, entry.f1);
- }
- return clusterIdByPoints;
- }
-
- private static void verifyClusteringResult(
- Map<DenseVector, Integer> clusterIdByPoints, List<List<Integer>> groups) {
- for (List<Integer> group : groups) {
- for (int i = 1; i < group.size(); i++) {
- assertEquals(
- clusterIdByPoints.get(DATA.get(group.get(0))),
- clusterIdByPoints.get(DATA.get(group.get(i))));
+ private static List<Set<DenseVector>> executeAndCollect(
+ Table output, String featureCol, String predictionCol) {
+ Map<Integer, Set<DenseVector>> map = new HashMap<>();
+ for (CloseableIterator<Row> it = output.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ DenseVector vector = (DenseVector) row.getField(featureCol);
+ int predict = (Integer) row.getField(predictionCol);
+ if (!map.containsKey(predict)) {
+ map.put(predict, new HashSet<>());
Review comment:
nits: `map.putIfAbsent(predict, new HashSet<>())`
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+
+import static org.apache.flink.table.api.Expressions.$;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private List<Row> trainData;
+ private List<Row> predictData;
+ private List<Row> expectedOutput;
+ private boolean isSaveLoad;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ schema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.DOUBLE())
+ .column("f1", DataTypes.of(DenseVector.class))
+ .column("f2", DataTypes.DOUBLE())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainData =
+ Arrays.asList(
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0));
+
+ predictData = trainData;
+
+ expectedOutput =
+ Arrays.asList(
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0, 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0, 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0, 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0, 10.0));
+
+ isSaveLoad = false;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testNaiveBayes() throws Exception {
Review comment:
nits: This test method is a bit broad here given that the class name is already `NaiveBayesTest` and every tests in this class is supposed to "test NaiveBayes". How about changing this to `testFitAndPredict`?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Double>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Double>>() {
+ @Override
+ public Tuple2<Vector, Double> map(Row row) throws Exception {
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ (Double) row.getField(labelCol));
+ }
+ });
+
+ DataStream<NaiveBayesModelData> naiveBayesModel =
+ input.flatMap(new FlattenFunction())
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce(
+ (ReduceFunction<Tuple4<Double, Integer, Double, Double>>)
+ (t0, t1) -> {
+ t0.f3 += t1.f3;
+ return t0;
+ })
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy(
+ (KeySelector<
+ Tuple4<
+ Double,
+ Integer,
+ Map<Double, Double>,
+ Double>,
+ Object>)
+ value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction())
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(smoothing));
+
+ NaiveBayesModel model =
+ new NaiveBayesModel()
+ .setModelData(NaiveBayesModelData.fromDataStream(tEnv, naiveBayesModel));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(StreamExecutionEnvironment env, String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value,
+ * weight.
+ */
+ private static class FlattenFunction
+ implements FlatMapFunction<
+ Tuple2<Vector, Double>, Tuple4<Double, Integer, Double, Double>> {
Review comment:
Would it be better to case `label` to `Integer` instead of `Double`, given that the label in `NaiveBayes` should always be `Integer`?
Note that the label value is used as the grouping key in the program. I am not sure if there will misclassification due to precision error.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Object.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelStream =
+ NaiveBayesModelData.toDataStream(tEnv, modelTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(
+ new PredictLabelFunction(featuresCol, broadcastModelKey)));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.toDataStream(tEnv, modelTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(StreamExecutionEnvironment env, String path)
+ throws IOException {
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ Source<NaiveBayesModelData, ?, ?> source =
+ FileSource.forRecordStreamFormat(
+ new NaiveBayesModelData.ModelDataStreamFormat(),
+ ReadWriteUtils.getDataPaths(path))
+ .build();
+ NaiveBayesModel model = ReadWriteUtils.loadStageParam(path);
+ DataStream<NaiveBayesModelData> modelData =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "modelData");
+ model.setModelData(NaiveBayesModelData.fromDataStream(tEnv, modelData));
+
+ return model;
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ @Override
+ public NaiveBayesModel setModelData(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ modelTable = inputs[0];
+ return this;
+ }
+
+ @Override
+ public Table[] getModelData() {
+ return new Table[] {modelTable};
+ }
+
+ private static class PredictLabelOperator
+ extends AbstractUdfStreamOperator<Row, PredictLabelFunction>
+ implements OneInputStreamOperator<Row, Row> {
+ public PredictLabelOperator(PredictLabelFunction userFunction) {
+ super(userFunction);
+ }
+
+ @Override
+ public void processElement(StreamRecord<Row> streamRecord) {
Review comment:
Would it be simpler to remove `PredictLabelFunction` and move its logic into `PredictLabelOperator::processElement`?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Double>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Double>>() {
+ @Override
+ public Tuple2<Vector, Double> map(Row row) throws Exception {
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ (Double) row.getField(labelCol));
+ }
+ });
+
+ DataStream<NaiveBayesModelData> naiveBayesModel =
Review comment:
nits: the name`naiveBayesModel` may indicate this is a `Model` while this is actually model data. How about renaming this to `modelData`?
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/util/StageTestUtils.java
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.util;
+
+import org.apache.flink.ml.api.core.Stage;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import java.lang.reflect.Method;
+import java.nio.file.Files;
+
+/** Utility methods for testing stages. */
+public class StageTestUtils {
+ /**
+ * Saves a stage to filesystem and reloads it with the static load() method a stage must
+ * implement.
+ */
+ public static <T extends Stage<T>> T saveAndReload(StreamExecutionEnvironment env, T stage)
+ throws Exception {
+ String tempDir = Files.createTempDirectory("").toString();
Review comment:
Nice!
Could you help make one more improvement by using `org.junit.rules.TemporaryFolder` to create temporary directory in Junit tests? This is the approach used in most (if not all) Flink tests (see `KafkaTestBase` for example). According to the Java doc of `TemporaryFolder`, this approach can delete folders after the test method finishes.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Double>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Double>>() {
+ @Override
+ public Tuple2<Vector, Double> map(Row row) throws Exception {
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ (Double) row.getField(labelCol));
+ }
+ });
+
+ DataStream<NaiveBayesModelData> naiveBayesModel =
+ input.flatMap(new FlattenFunction())
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce(
+ (ReduceFunction<Tuple4<Double, Integer, Double, Double>>)
+ (t0, t1) -> {
+ t0.f3 += t1.f3;
+ return t0;
+ })
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy(
+ (KeySelector<
+ Tuple4<
+ Double,
+ Integer,
+ Map<Double, Double>,
+ Double>,
+ Object>)
+ value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction())
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(smoothing));
+
+ NaiveBayesModel model =
+ new NaiveBayesModel()
+ .setModelData(NaiveBayesModelData.fromDataStream(tEnv, naiveBayesModel));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(StreamExecutionEnvironment env, String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value,
+ * weight.
+ */
+ private static class FlattenFunction
+ implements FlatMapFunction<
+ Tuple2<Vector, Double>, Tuple4<Double, Integer, Double, Double>> {
+ @Override
+ public void flatMap(
+ Tuple2<Vector, Double> value,
+ Collector<Tuple4<Double, Integer, Double, Double>> collector) {
+ Preconditions.checkNotNull(value.f1);
+ for (int i = 0; i < value.f0.size(); i++) {
+ collector.collect(new Tuple4<>(value.f1, i, value.f0.get(i), 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and
+ * feature column index.
+ */
+ private static class ValueMapFunction
+ implements AggregateFunction<
+ Tuple4<Double, Integer, Double, Double>,
+ Tuple3<Double, Integer, Map<Double, Double>>,
+ Tuple4<Double, Integer, Map<Double, Double>, Double>> {
+
+ @Override
+ public Tuple3<Double, Integer, Map<Double, Double>> createAccumulator() {
+ return new Tuple3<>(0., -1, new HashMap<>());
+ }
+
+ @Override
+ public Tuple3<Double, Integer, Map<Double, Double>> add(
+ Tuple4<Double, Integer, Double, Double> value,
+ Tuple3<Double, Integer, Map<Double, Double>> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Double, Integer, Map<Double, Double>, Double> getResult(
+ Tuple3<Double, Integer, Map<Double, Double>> acc) {
+ double weightSum = acc.f2.values().stream().mapToDouble(f -> f).sum();
Review comment:
nits: it seems a bit better to use the approach explained in https://stackoverflow.com/questions/30125296/how-to-sum-a-list-of-integers-with-java-streams
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasLabelCol.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.StringParam;
+import org.apache.flink.ml.param.WithParams;
+
+/**
+ * Param of the name of the label column in the input table.
Review comment:
nits: If we follow the pattern of other param subclasses, the comment could be `Interface for the shared labelCol param.`
The param <T> is specified but not explained in this comment. We could just remove it for consistency with other params.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755746528
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Double>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Double>>() {
+ @Override
+ public Tuple2<Vector, Double> map(Row row) throws Exception {
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ (Double) row.getField(labelCol));
+ }
+ });
+
+ DataStream<NaiveBayesModelData> naiveBayesModel =
+ input.flatMap(new FlattenFunction())
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce(
+ (ReduceFunction<Tuple4<Double, Integer, Double, Double>>)
+ (t0, t1) -> {
+ t0.f3 += t1.f3;
+ return t0;
+ })
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy(
+ (KeySelector<
+ Tuple4<
+ Double,
+ Integer,
+ Map<Double, Double>,
+ Double>,
+ Object>)
+ value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction())
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(smoothing));
+
+ NaiveBayesModel model =
+ new NaiveBayesModel()
+ .setModelData(NaiveBayesModelData.fromDataStream(tEnv, naiveBayesModel));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(StreamExecutionEnvironment env, String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value,
+ * weight.
+ */
+ private static class FlattenFunction
+ implements FlatMapFunction<
+ Tuple2<Vector, Double>, Tuple4<Double, Integer, Double, Double>> {
+ @Override
+ public void flatMap(
+ Tuple2<Vector, Double> value,
+ Collector<Tuple4<Double, Integer, Double, Double>> collector) {
+ Preconditions.checkNotNull(value.f1);
+ for (int i = 0; i < value.f0.size(); i++) {
+ collector.collect(new Tuple4<>(value.f1, i, value.f0.get(i), 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and
+ * feature column index.
+ */
+ private static class ValueMapFunction
+ implements AggregateFunction<
+ Tuple4<Double, Integer, Double, Double>,
+ Tuple3<Double, Integer, Map<Double, Double>>,
+ Tuple4<Double, Integer, Map<Double, Double>, Double>> {
+
+ @Override
+ public Tuple3<Double, Integer, Map<Double, Double>> createAccumulator() {
+ return new Tuple3<>(0., -1, new HashMap<>());
+ }
+
+ @Override
+ public Tuple3<Double, Integer, Map<Double, Double>> add(
+ Tuple4<Double, Integer, Double, Double> value,
+ Tuple3<Double, Integer, Map<Double, Double>> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Double, Integer, Map<Double, Double>, Double> getResult(
+ Tuple3<Double, Integer, Map<Double, Double>> acc) {
+ double weightSum = acc.f2.values().stream().mapToDouble(f -> f).sum();
+ return new Tuple4<>(acc.f0, acc.f1, acc.f2, weightSum);
+ }
+
+ @Override
+ public Tuple3<Double, Integer, Map<Double, Double>> merge(
+ Tuple3<Double, Integer, Map<Double, Double>> acc0,
+ Tuple3<Double, Integer, Map<Double, Double>> acc1) {
+ Preconditions.checkArgument(acc0.f1 != -1);
+ acc0.f2.putAll(acc1.f2);
+ return acc0;
+ }
+ }
+
+ /** Function to aggregate maps under the same label into arrays. array len = featureSize. */
+ private static class MapArrayFunction
Review comment:
OK. I'll do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755988672
##########
File path: flink-ml-api/src/main/java/org/apache/flink/ml/linalg/BLAS.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.linalg;
+
+import org.apache.flink.util.Preconditions;
+
+/** A utility class that provides BLAS routines over matrices and vectors. */
+public class BLAS {
+ /**
+ * For level-2 and level-3 routines, we use the native BLAS.
+ *
Review comment:
Can you update the comments here?
Also, why do we put BLAS in `flink-ml-api`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755751841
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.Map;
+
+/** The model data of {@link NaiveBayesModel}. Also provides classes to save/load model data. */
+public class NaiveBayesModelData implements Serializable {
+ private static final long serialVersionUID = 3919917903722286395L;
+ public final Map<Double, Double>[][] theta;
+ public final double[] piArray;
+ public final double[] labels;
+
+ // Empty constructor is used when Kyro deserializes loaded model data.
+ public NaiveBayesModelData() {
+ this(null, null, null);
+ }
+
+ public NaiveBayesModelData(Map<Double, Double>[][] theta, double[] piArray, double[] labels) {
+ this.theta = theta;
+ this.piArray = piArray;
+ this.labels = labels;
+ }
+
+ public static Table fromDataStream(
Review comment:
Sure. I'll make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755746409
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Double>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Double>>() {
+ @Override
+ public Tuple2<Vector, Double> map(Row row) throws Exception {
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ (Double) row.getField(labelCol));
+ }
+ });
+
+ DataStream<NaiveBayesModelData> naiveBayesModel =
+ input.flatMap(new FlattenFunction())
Review comment:
OK. I'll fix it and add clearer comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
lindong28 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r756947787
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/util/StageTestUtils.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.util;
+
+import org.apache.flink.ml.api.core.Stage;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import org.junit.rules.TemporaryFolder;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+
+/** Utility methods for testing stages. */
+public class StageTestUtils {
+ private static final TemporaryFolder tempFolder = new TemporaryFolder();
+
+ static {
+ try {
+ tempFolder.create();
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ /**
+ * Saves a stage to filesystem and reloads it with the static load() method a stage must
Review comment:
nits: "a stage must implement" seems irrelevant here. Maybe change it to `Saves a stage to filesystem and reloads it by invoking the static load() method of the given stage`.
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Table trainTable;
+ private Table predictTable;
+ private Map<Vector, Integer> expectedOutput;
+ private NaiveBayes estimator;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ Schema trainSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .column("f1", DataTypes.INT())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 0.), 11),
+ Row.of(Vectors.dense(1, 0), 10),
+ Row.of(Vectors.dense(1, 1.), 10))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ trainSchema)
+ .as("features", "label");
+
+ Schema predictSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 1.)),
+ Row.of(Vectors.dense(0, 0.)),
+ Row.of(Vectors.dense(1, 0)),
+ Row.of(Vectors.dense(1, 1.)))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ predictSchema)
+ .as("features");
+
+ expectedOutput =
+ new HashMap<Vector, Integer>() {
+ {
+ put(Vectors.dense(0, 1.), 11);
+ put(Vectors.dense(0, 0.), 11);
+ put(Vectors.dense(1, 0.), 10);
+ put(Vectors.dense(1, 1.), 10);
+ }
+ };
+
+ estimator =
+ new NaiveBayes()
+ .setSmoothing(1.0)
+ .setFeaturesCol("features")
+ .setLabelCol("label")
+ .setPredictionCol("predict")
+ .setModelType("multinomial");
+ }
+
+ /**
+ * Executes a given table and collect its results. Results are returned as a map whose key is
+ * the feature, value is the prediction result.
+ *
+ * @param table A table to be executed and to have its result collected
+ * @param featuresCol Name of the column in the table that contains the features
+ * @param predictionCol Name of the column in the table that contains the prediction result
+ * @return A map containing the collected results
+ */
+ private static Map<Vector, Integer> executeAndCollect(
+ Table table, String featuresCol, String predictionCol) {
+ Map<Vector, Integer> map = new HashMap<>();
+ for (CloseableIterator<Row> it = table.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ map.put(
+ (Vector) row.getField(featuresCol),
+ ((Number) row.getField(predictionCol)).intValue());
+ }
+
+ return map;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testFitAndPredict() throws Exception {
+ NaiveBayesModel model = estimator.fit(trainTable);
+ Table outputTable = model.transform(predictTable)[0];
+ Map<Vector, Integer> actualOutput =
+ executeAndCollect(outputTable, model.getFeaturesCol(), model.getPredictionCol());
+ assertEquals(expectedOutput, actualOutput);
+ }
+
+ @Test(expected = Exception.class)
+ public void testPredictUnseenFeature() throws Exception {
Review comment:
Do we expect the Exception message to provide information regarding the "unseen feature"? If so, would it be better to also check the message, similar to the existing test such as `StageTest::testParamWithNullDefault`?
Same for other tests.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Integer>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Integer>>() {
+ @Override
+ public Tuple2<Vector, Integer> map(Row row) throws Exception {
+ Number number = (Number) row.getField(labelCol);
+ Preconditions.checkArgument(
+ number.intValue() == number.doubleValue());
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ number.intValue());
+ }
+ });
+
+ DataStream<NaiveBayesModelData> modelData =
+ input.flatMap(new ExtractFeatureFunction())
+ .keyBy(
+ (KeySelector<Tuple4<Integer, Integer, Double, Double>, Object>)
+ value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce(
+ (ReduceFunction<Tuple4<Integer, Integer, Double, Double>>)
+ (t0, t1) ->
+ new Tuple4<>(t0.f0, t0.f1, t0.f2, t0.f3 + t1.f3))
+ .keyBy(
+ (KeySelector<Tuple4<Integer, Integer, Double, Double>, Object>)
+ value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new GenerateFeatureWeightMapFunction())
+ .keyBy(
+ (KeySelector<
+ Tuple4<
+ Integer,
+ Integer,
+ Map<Double, Double>,
+ Double>,
+ Object>)
+ value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new AggregateIntoArrayFunction())
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(smoothing));
+
+ NaiveBayesModel model =
+ new NaiveBayesModel()
+ .setModelData(NaiveBayesModelData.getModelDataTable(tEnv, modelData));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(StreamExecutionEnvironment env, String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to extract feature values from input rows.
+ *
+ * <p>Output records are tuples with the following fields in order:
+ *
+ * <ul>
+ * <li>label value
+ * <li>feature column index
+ * <li>feature value
+ * <li>weight
+ * </ul>
+ */
+ private static class ExtractFeatureFunction
+ implements FlatMapFunction<
+ Tuple2<Vector, Integer>, Tuple4<Integer, Integer, Double, Double>> {
+ @Override
+ public void flatMap(
+ Tuple2<Vector, Integer> value,
+ Collector<Tuple4<Integer, Integer, Double, Double>> collector) {
+ Preconditions.checkNotNull(value.f1);
+ for (int i = 0; i < value.f0.size(); i++) {
+ collector.collect(new Tuple4<>(value.f1, i, value.f0.get(i), 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function that aggregates entries of feature value and weight into maps.
+ *
+ * <p>Input records should have the same label value and feature column index.
+ *
+ * <p>Input records are tuples with the following fields in order:
+ *
+ * <ul>
+ * <li>label value
+ * <li>feature column index
+ * <li>feature value
+ * <li>weight
+ * </ul>
+ *
+ * <p>Output records are tuples with the following fields in order:
+ *
+ * <ul>
+ * <li>label value
+ * <li>feature column index
+ * <li>map of (feature value, weight)
+ * </ul>
+ */
+ private static class GenerateFeatureWeightMapFunction
+ implements AggregateFunction<
+ Tuple4<Integer, Integer, Double, Double>,
+ Tuple3<Integer, Integer, Map<Double, Double>>,
+ Tuple4<Integer, Integer, Map<Double, Double>, Double>> {
+
+ @Override
+ public Tuple3<Integer, Integer, Map<Double, Double>> createAccumulator() {
+ return new Tuple3<>(0, -1, new HashMap<>());
+ }
+
+ @Override
+ public Tuple3<Integer, Integer, Map<Double, Double>> add(
+ Tuple4<Integer, Integer, Double, Double> value,
+ Tuple3<Integer, Integer, Map<Double, Double>> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Integer, Integer, Map<Double, Double>, Double> getResult(
+ Tuple3<Integer, Integer, Map<Double, Double>> acc) {
+ double weightSum = acc.f2.values().stream().mapToDouble(Double::doubleValue).sum();
+ return new Tuple4<>(acc.f0, acc.f1, acc.f2, weightSum);
+ }
+
+ @Override
+ public Tuple3<Integer, Integer, Map<Double, Double>> merge(
+ Tuple3<Integer, Integer, Map<Double, Double>> acc0,
+ Tuple3<Integer, Integer, Map<Double, Double>> acc1) {
+ Preconditions.checkArgument(acc0.f1 != -1);
+ acc0.f2.putAll(acc1.f2);
+ return acc0;
+ }
+ }
+
+ /**
+ * Function that aggregates maps under the same label into arrays.
+ *
+ * <p>Length of the generated array equals to the number of feature columns.
+ *
+ * <p>Input records are tuples with the following fields in order:
+ *
+ * <ul>
+ * <li>label value
+ * <li>feature column index
+ * <li>map of (feature value, weight)
Review comment:
It seems that the 4th field of the input is missing?
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Table trainTable;
+ private Table predictTable;
+ private Map<Vector, Integer> expectedOutput;
+ private NaiveBayes estimator;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ Schema trainSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .column("f1", DataTypes.INT())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 0.), 11),
+ Row.of(Vectors.dense(1, 0), 10),
+ Row.of(Vectors.dense(1, 1.), 10))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ trainSchema)
+ .as("features", "label");
+
+ Schema predictSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 1.)),
+ Row.of(Vectors.dense(0, 0.)),
+ Row.of(Vectors.dense(1, 0)),
+ Row.of(Vectors.dense(1, 1.)))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ predictSchema)
+ .as("features");
+
+ expectedOutput =
+ new HashMap<Vector, Integer>() {
+ {
+ put(Vectors.dense(0, 1.), 11);
+ put(Vectors.dense(0, 0.), 11);
+ put(Vectors.dense(1, 0.), 10);
+ put(Vectors.dense(1, 1.), 10);
+ }
+ };
+
+ estimator =
+ new NaiveBayes()
+ .setSmoothing(1.0)
+ .setFeaturesCol("features")
+ .setLabelCol("label")
+ .setPredictionCol("predict")
+ .setModelType("multinomial");
+ }
+
+ /**
+ * Executes a given table and collect its results. Results are returned as a map whose key is
+ * the feature, value is the prediction result.
+ *
+ * @param table A table to be executed and to have its result collected
+ * @param featuresCol Name of the column in the table that contains the features
+ * @param predictionCol Name of the column in the table that contains the prediction result
+ * @return A map containing the collected results
+ */
+ private static Map<Vector, Integer> executeAndCollect(
+ Table table, String featuresCol, String predictionCol) {
+ Map<Vector, Integer> map = new HashMap<>();
+ for (CloseableIterator<Row> it = table.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ map.put(
+ (Vector) row.getField(featuresCol),
+ ((Number) row.getField(predictionCol)).intValue());
+ }
+
+ return map;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testFitAndPredict() throws Exception {
Review comment:
Could you help update this test to use non-default column names for features, labels and predictions? I hope to do this test for every algorithm to make sure the user-specified columns names are enforced.
This could be similar to `KMeansTest::testFeaturePredictionParam`.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
Review comment:
Do you know what could go wrong if we don't specify `serialVersionUID` for `NaiveBayesModel` and `NaiveBayesModelData`? Spark does not seem to explicitly specify UID for its Estimator/Model subclasses.
If there is no known issue yet, maybe we should remove this UID for simplicity.
If there is known impact, we will need to add this UID for `NaiveBayes`, `KMeans` and `KMeansModel` etc.
##########
File path: flink-ml-api/src/main/java/org/apache/flink/ml/linalg/BLAS.java
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.linalg;
+
+import org.apache.flink.util.Preconditions;
+
+/** A utility class that provides BLAS routines over matrices and vectors. */
+public class BLAS {
+ /** For level-1 routines, we use Java implementation. */
+ private static final dev.ludovic.netlib.BLAS F2J_BLAS =
+ dev.ludovic.netlib.JavaBLAS.getInstance();
+
+ /** y += a * x . */
+ public static void axpy(double a, double[] x, double[] y) {
Review comment:
In the future will we need to support `axpy(a: Double, x: SparseVector, y: DenseVector)`? If so, should we make this `axpy(a: Double, x: DenseVector, y: DenseVector)`?
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/util/StageTestUtils.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.util;
+
+import org.apache.flink.ml.api.core.Stage;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import org.junit.rules.TemporaryFolder;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+
+/** Utility methods for testing stages. */
+public class StageTestUtils {
+ private static final TemporaryFolder tempFolder = new TemporaryFolder();
+
+ static {
+ try {
+ tempFolder.create();
Review comment:
Why do we need to call `tempFolder.create`?
The Java doc of `tempFolder.create` says: `for testing purposes only. Do not use`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r762744664
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The model data of {@link NaiveBayesModel}.
+ *
+ * <p>This class also provides methods to convert model data between Table and Datastream, and
+ * classes to save/load model data.
+ */
+public class NaiveBayesModelData implements Serializable {
+ public final Map<Double, Double>[][] theta;
+ public final double[] piArray;
+ public final int[] labels;
+
+ public NaiveBayesModelData(Map<Double, Double>[][] theta, double[] piArray, int[] labels) {
+ this.theta = theta;
+ this.piArray = piArray;
+ this.labels = labels;
+ }
+
+ /** Converts the provided modelData Datastream into corresponding Table. */
+ public static Table getModelDataTable(DataStream<NaiveBayesModelData> stream) {
+ StreamTableEnvironment tEnv =
+ StreamTableEnvironment.create(stream.getExecutionEnvironment());
+ return tEnv.fromDataStream(stream);
+ }
+
+ /** Converts the provided modelData Table into corresponding DataStream. */
+ public static DataStream<NaiveBayesModelData> getModelDataStream(Table table) {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) table).getTableEnvironment();
+ return tEnv.toDataStream(table)
+ .map(
+ (MapFunction<Row, NaiveBayesModelData>)
+ row -> (NaiveBayesModelData) row.getField("f0"));
+ }
+
+ /** Encoder for the {@link NaiveBayesModelData}. */
+ public static class ModelDataEncoder implements Encoder<NaiveBayesModelData> {
+ @Override
+ public void encode(NaiveBayesModelData modelData, OutputStream outputStream) {
+ Output output = new Output(outputStream);
+
+ output.writeInt(modelData.labels.length);
Review comment:
nits: can you use Flink's serializer? Similar as` LogisticRegressionModelData#ModelDataEncoder`
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Integer.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelDataTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelDataStream =
+ NaiveBayesModelData.getModelDataStream(modelDataTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelDataStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(featuresCol, broadcastModelKey));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
Review comment:
nits: can you checkout `LogisticRegressionModelData#save/load`?
I am not sure which one is better, but I think we should follow same programming pattern.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Integer.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelDataTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelDataStream =
+ NaiveBayesModelData.getModelDataStream(modelDataTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelDataStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(featuresCol, broadcastModelKey));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
Review comment:
nits: can we use `Collections.singletonMap(broadcastModelKey, modelDataStream)` instead?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Integer.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelDataTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelDataStream =
+ NaiveBayesModelData.getModelDataStream(modelDataTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelDataStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(featuresCol, broadcastModelKey));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.getModelDataStream(modelDataTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(StreamExecutionEnvironment env, String path)
+ throws IOException {
+ Source<NaiveBayesModelData, ?, ?> source =
+ FileSource.forRecordStreamFormat(
+ new NaiveBayesModelData.ModelDataStreamFormat(),
+ ReadWriteUtils.getDataPaths(path))
+ .build();
+ NaiveBayesModel model = ReadWriteUtils.loadStageParam(path);
+ DataStream<NaiveBayesModelData> modelData =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "modelData");
+ model.setModelData(NaiveBayesModelData.getModelDataTable(modelData));
+
+ return model;
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ @Override
+ public NaiveBayesModel setModelData(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ modelDataTable = inputs[0];
+ return this;
+ }
+
+ @Override
+ public Table[] getModelData() {
+ return new Table[] {modelDataTable};
+ }
+
+ private static class PredictLabelOperator
+ extends AbstractUdfStreamOperator<Row, AbstractRichFunction>
+ implements OneInputStreamOperator<Row, Row> {
+ private final String featuresCol;
+ private final String broadcastModelKey;
+
+ public PredictLabelOperator(String featuresCol, String broadcastModelKey) {
+ super(new AbstractRichFunction() {});
+ this.featuresCol = featuresCol;
+ this.broadcastModelKey = broadcastModelKey;
+ }
+
+ @Override
+ public void processElement(StreamRecord<Row> streamRecord) {
+ Row inputRow = streamRecord.getValue();
+ NaiveBayesModelData modelData =
Review comment:
nits: can you cache the `modelData` instead of accessing repeatly from the broadcast context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r765397328
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Integer>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Integer>>() {
+ @Override
+ public Tuple2<Vector, Integer> map(Row row) throws Exception {
+ Number number = (Number) row.getField(labelCol);
+ Preconditions.checkArgument(
+ number.intValue() == number.doubleValue());
Review comment:
OK. I'll add one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r764639459
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Integer>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Integer>>() {
+ @Override
+ public Tuple2<Vector, Integer> map(Row row) throws Exception {
+ Number number = (Number) row.getField(labelCol);
+ Preconditions.checkArgument(
+ number.intValue() == number.doubleValue());
Review comment:
I wonder whether this check is necessary. If the number is null then the program will soon fail with NullPointerException. Adding an additional check seems not to change this behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] gaoyunhaii commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r764955406
##########
File path: flink-ml-core/src/main/java/org/apache/flink/ml/linalg/BLAS.java
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.linalg;
+
+import org.apache.flink.util.Preconditions;
+
+/** A utility class that provides BLAS routines over matrices and vectors. */
+public class BLAS {
+ /** For level-1 routines, we use Java implementation. */
+ private static final dev.ludovic.netlib.BLAS F2J_BLAS =
Review comment:
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala
Sorry I do not see `F2J_BLAS` in the master spark code? Also, by googling it seems F2J_BLAS is referring to `com.github.fommil.netlib.F2jBLAS`, which seems not used here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750848148
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
+ return;
+ }
+
+ for (int i = 0; i < featureSize; i++) {
+ Object feature = row.getField(featureColNames[i]);
+ if (feature == null) {
+ continue;
+ }
+ collector.collect(new Tuple4<>(label, i, feature, 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and feature column index.
+ */
+ private static class ValueMapFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Object, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>> {
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> createAccumulator() {
+ return new Tuple4<>(new Object(), -1, new HashMap<>(), 0.);
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> add(
+ Tuple4<Object, Integer, Object, Double> value,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ acc.f3 += value.f3;
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> getResult(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> merge(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc0,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc1) {
+ if (acc0.f0.equals(new Object())) {
+ acc0.f0 = acc1.f0;
+ }
+ if (acc0.f1 == -1) {
+ acc0.f1 = acc1.f1;
+ }
+ for (Map.Entry<Object, Double> entry: acc1.f2.entrySet()) {
+ acc0.f2.put(entry.getKey(), entry.getValue());
+ }
+ acc0.f3 += acc1.f3;
+
+ return acc0;
+ }
+ }
+
+ /**
+ * Function to aggregate maps under the same label into arrays. array len = featureSize
+ */
+ private static class MapArrayFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple3<Object, Double[], Map <Object, Double>[]>,
+ Tuple3<Object, Double[], Map <Object, Double>[]>> {
+ private final int featureSize;
+
+ private MapArrayFunction(int featureSize) {
+ this.featureSize = featureSize;
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> createAccumulator() {
+ Double[] weightSum = new Double[featureSize];
+ Arrays.fill(weightSum, 0.);
+ return new Tuple3<>(new Object(), weightSum, new HashMap[featureSize]);
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> add(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> value,
+ Tuple3<Object, Double[], Map<Object, Double>[]> acc) {
+ acc.f0 = value.f0;
+ acc.f1[value.f1] = value.f3;
+ acc.f2[value.f1] = value.f2;
+
+ return acc;
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> getResult(Tuple3<Object, Double[], Map<Object, Double>[]> acc) {
+ return acc;
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> merge(
+ Tuple3<Object, Double[], Map<Object, Double>[]> acc0,
+ Tuple3<Object, Double[], Map<Object, Double>[]> acc1) {
+ if (acc0.f0.equals(new Object())) {
+ acc0.f0 = acc1.f0;
+ }
+ for (int i = 0; i < featureSize; i++) {
+ acc0.f1[i] += acc1.f1[i];
+ if (acc1.f2[i] != null) {
+ acc0.f2[i] = acc1.f2[i];
+ }
+ }
+
+ return acc0;
+ }
+ }
+
+ /**
+ * Function to generate Naive Bayes model data.
+ */
+ private static class GenerateModelFunction implements AllWindowFunction<
+ Tuple3<Object, Double[], Map<Object, Double>[]>,
+ NaiveBayesModelData,
+ TimeWindow
+ > {
+ private final int featureSize;
+ private final double smoothing;
+ private final String[] featureColNames;
+
+ GenerateModelFunction(double smoothing, String[] featureColNames) {
+ this.smoothing = smoothing;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void apply(TimeWindow timeWindow,
+ Iterable<Tuple3<Object, Double[], Map<Object, Double>[]>> values,
+ Collector<NaiveBayesModelData> out) {
+ double[] numDocs = new double[featureSize];
+ ArrayList <Tuple3 <Object, Double[], Map <Object, Double>[]>> modelArray = new ArrayList <>();
+ HashSet <Object>[] categoryNumbers = new HashSet[featureSize];
Review comment:
'll run `mvn spotless:apply` to format the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750876450
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesParams.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.ml.param.DoubleParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.shared.colname.HasFeatureCols;
+import org.apache.flink.ml.param.shared.colname.HasLabelCol;
+import org.apache.flink.ml.param.shared.colname.HasPredictionCol;
+
+/**
+ * Parameters of naive bayes training process.
+ */
+public interface NaiveBayesParams<T> extends
+ HasFeatureCols<T>,
+ HasLabelCol<T>,
+ HasPredictionCol<T> {
+ Param<Double> SMOOTHING =
+ new DoubleParam(
+ "smoothing",
+ "the smoothing factor",
+ 0.0,
Review comment:
I think either value makes sense. But as Spark chooss 1.0 let's also follow this practice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r753076594
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.api.Expressions.$;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private Row[] trainData;
+ private Row[] predictData;
+ private Row[] expectedOutput;
+ private String featuresCol;
+ private String labelCol;
+ private String predictCol;
+ private String modelType;
+ private double smoothing;
+ private boolean isSaveLoad;
+ private String errorMessage;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ schema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.DOUBLE())
+ .column("f1", DataTypes.of(DenseVector.class))
+ .column("f2", DataTypes.DOUBLE())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+
+ predictData = trainData;
+
+ expectedOutput =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0, 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0, 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0, 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0, 10.0)
+ };
+
+ featuresCol = "features";
Review comment:
OK. I'll change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755727278
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Double>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Double>>() {
+ @Override
+ public Tuple2<Vector, Double> map(Row row) throws Exception {
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ (Double) row.getField(labelCol));
+ }
+ });
+
+ DataStream<NaiveBayesModelData> naiveBayesModel =
Review comment:
Yes, `modeData` should be more clear. I'll make this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755747903
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasLabelCol.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.param;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.StringParam;
+import org.apache.flink.ml.param.WithParams;
+
+/**
+ * Param of the name of the label column in the input table.
Review comment:
OK. I'll fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755759816
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Double>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Double>>() {
+ @Override
+ public Tuple2<Vector, Double> map(Row row) throws Exception {
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ (Double) row.getField(labelCol));
+ }
+ });
+
+ DataStream<NaiveBayesModelData> naiveBayesModel =
+ input.flatMap(new FlattenFunction())
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce(
+ (ReduceFunction<Tuple4<Double, Integer, Double, Double>>)
+ (t0, t1) -> {
+ t0.f3 += t1.f3;
+ return t0;
+ })
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy(
+ (KeySelector<
+ Tuple4<
+ Double,
+ Integer,
+ Map<Double, Double>,
+ Double>,
+ Object>)
+ value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction())
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(smoothing));
+
+ NaiveBayesModel model =
+ new NaiveBayesModel()
+ .setModelData(NaiveBayesModelData.fromDataStream(tEnv, naiveBayesModel));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(StreamExecutionEnvironment env, String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value,
+ * weight.
+ */
+ private static class FlattenFunction
+ implements FlatMapFunction<
+ Tuple2<Vector, Double>, Tuple4<Double, Integer, Double, Double>> {
+ @Override
+ public void flatMap(
+ Tuple2<Vector, Double> value,
+ Collector<Tuple4<Double, Integer, Double, Double>> collector) {
+ Preconditions.checkNotNull(value.f1);
+ for (int i = 0; i < value.f0.size(); i++) {
+ collector.collect(new Tuple4<>(value.f1, i, value.f0.get(i), 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and
+ * feature column index.
+ */
+ private static class ValueMapFunction
+ implements AggregateFunction<
+ Tuple4<Double, Integer, Double, Double>,
+ Tuple3<Double, Integer, Map<Double, Double>>,
+ Tuple4<Double, Integer, Map<Double, Double>, Double>> {
+
+ @Override
+ public Tuple3<Double, Integer, Map<Double, Double>> createAccumulator() {
+ return new Tuple3<>(0., -1, new HashMap<>());
+ }
+
+ @Override
+ public Tuple3<Double, Integer, Map<Double, Double>> add(
+ Tuple4<Double, Integer, Double, Double> value,
+ Tuple3<Double, Integer, Map<Double, Double>> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Double, Integer, Map<Double, Double>, Double> getResult(
+ Tuple3<Double, Integer, Map<Double, Double>> acc) {
+ double weightSum = acc.f2.values().stream().mapToDouble(f -> f).sum();
Review comment:
OK. I'll make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755671196
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/clustering/KMeansTest.java
##########
@@ -83,43 +97,19 @@ public void before() {
dataTable = tEnv.fromDataStream(env.fromCollection(DATA), schema).as("features");
}
- // Executes the graph and returns a map which maps points to clusterId.
- private static Map<DenseVector, Integer> executeAndCollect(
- Table output, String featureCol, String predictionCol) throws Exception {
- StreamTableEnvironment tEnv =
- (StreamTableEnvironment) ((TableImpl) output).getTableEnvironment();
-
- DataStream<Tuple2<DenseVector, Integer>> stream =
- tEnv.toDataStream(output)
- .map(
- new MapFunction<Row, Tuple2<DenseVector, Integer>>() {
- @Override
- public Tuple2<DenseVector, Integer> map(Row row) {
- return Tuple2.of(
- (DenseVector) row.getField(featureCol),
- (Integer) row.getField(predictionCol));
- }
- });
-
- List<Tuple2<DenseVector, Integer>> pointsWithClusterId =
- IteratorUtils.toList(stream.executeAndCollect());
-
- Map<DenseVector, Integer> clusterIdByPoints = new HashMap<>();
- for (Tuple2<DenseVector, Integer> entry : pointsWithClusterId) {
- clusterIdByPoints.put(entry.f0, entry.f1);
- }
- return clusterIdByPoints;
- }
-
- private static void verifyClusteringResult(
- Map<DenseVector, Integer> clusterIdByPoints, List<List<Integer>> groups) {
- for (List<Integer> group : groups) {
- for (int i = 1; i < group.size(); i++) {
- assertEquals(
- clusterIdByPoints.get(DATA.get(group.get(0))),
- clusterIdByPoints.get(DATA.get(group.get(i))));
+ private static List<Set<DenseVector>> executeAndCollect(
+ Table output, String featureCol, String predictionCol) {
+ Map<Integer, Set<DenseVector>> map = new HashMap<>();
+ for (CloseableIterator<Row> it = output.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ DenseVector vector = (DenseVector) row.getField(featureCol);
+ int predict = (Integer) row.getField(predictionCol);
+ if (!map.containsKey(predict)) {
+ map.put(predict, new HashSet<>());
Review comment:
OK. I'll fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755673036
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/clustering/KMeansTest.java
##########
@@ -158,10 +148,11 @@ public void testFeaturePredictionParam() throws Exception {
assertEquals(
Arrays.asList("test_feature", "test_prediction"),
output.getResolvedSchema().getColumnNames());
- Map<DenseVector, Integer> clusterIdByPoints =
- executeAndCollect(output, kmeans.getFeaturesCol(), kmeans.getPredictionCol());
- verifyClusteringResult(
- clusterIdByPoints, Arrays.asList(Arrays.asList(0, 1, 2), Arrays.asList(3, 4, 5)));
+ assertTrue(
+ CollectionUtils.isEqualCollection(
+ executeAndCollect(
Review comment:
OK. I'll fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r756544648
##########
File path: flink-ml-api/src/main/java/org/apache/flink/ml/linalg/BLAS.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.linalg;
+
+import org.apache.flink.util.Preconditions;
+
+/** A utility class that provides BLAS routines over matrices and vectors. */
+public class BLAS {
+ /**
+ * For level-2 and level-3 routines, we use the native BLAS.
+ *
Review comment:
Comments have been updated.
I think that BLAS should be placed in the same package as other linear algorithm classes like `Vector`. Given that `Vector` currently lies in `flink-ml-api`, I also placed `BLAS` there. We can discuss offline if we need to change the placement of such packages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r748081703
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
+ return;
+ }
+
+ for (int i = 0; i < featureSize; i++) {
+ Object feature = row.getField(featureColNames[i]);
+ if (feature == null) {
+ continue;
+ }
+ collector.collect(new Tuple4<>(label, i, feature, 1.0));
Review comment:
should we support weighted naive bayes or just remove 1.0 here?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
Review comment:
Shall we use different param classes for `NaiveBayes` and the `NaiveBayesModel`?
it may be confusing to users when call `new NaiveBayesModel().setSmoothing(...)` since `smoothing` is useless during the prediction.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/common/datastream/EndOfStreamWindows.java
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.triggers.EventTimeTrigger;
+import org.apache.flink.streaming.api.windowing.triggers.Trigger;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+
+import java.util.Collection;
+import java.util.Collections;
+
+/**
+ * A {@link WindowAssigner} that windows elements into windows based on the timestamp of the
+ * elements. Windows cannot overlap.
+ *
+ * <p>For example, in order to window into windows of 1 minute:
+ *
+ * <pre>{@code
+ * DataStream<Tuple2<String, Integer>> in = ...;
+ * KeyedStream<Tuple2<String, Integer>, String> keyed = in.keyBy(...);
+ * WindowedStream<Tuple2<String, Integer>, String, TimeWindow> windowed =
+ * keyed.window(TumblingEventTimeWindows.of(Time.minutes(1)));
+ * }</pre>
+ */
+@PublicEvolving
+public class EndOfStreamWindows extends WindowAssigner<Object, TimeWindow> {
+
+ private static final EndOfStreamWindows INSTANCE = new EndOfStreamWindows();
+
+ private EndOfStreamWindows() {}
+
+ public static EndOfStreamWindows get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Collection<TimeWindow> assignWindows(
+ Object element, long timestamp, WindowAssignerContext context) {
+ return Collections.singletonList(new TimeWindow(Long.MIN_VALUE, Long.MAX_VALUE));
Review comment:
why do you create a new TimeWindow on each call?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.ObjectArrayTypeInfo;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.Map;
+
+/**
+ * The model data of {@link NaiveBayesModel}.
+ */
+public class NaiveBayesModelData implements Serializable {
+ private static final long serialVersionUID = 3919917903722286395L;
+ public final String[] featureNames;
+ public final Map<Object, Double>[][] theta;
+ public final double[] piArray;
+ public final Object[] label;
+
+ public static final Schema SCHEMA =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(NaiveBayesModelData.class))
+ .build();
+
+ public NaiveBayesModelData(String[] featureNames, Map<Object, Double>[][] theta, double[] piArray, Object[] label) {
+ this.featureNames = featureNames;
+ this.theta = theta;
+ this.piArray = piArray;
+ this.label = label;
+ }
+
+ /** Encoder for the KMeans model data. */
Review comment:
update the comment here.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.ObjectArrayTypeInfo;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.Map;
+
+/**
+ * The model data of {@link NaiveBayesModel}.
+ */
+public class NaiveBayesModelData implements Serializable {
+ private static final long serialVersionUID = 3919917903722286395L;
+ public final String[] featureNames;
+ public final Map<Object, Double>[][] theta;
+ public final double[] piArray;
+ public final Object[] label;
+
+ public static final Schema SCHEMA =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(NaiveBayesModelData.class))
+ .build();
+
+ public NaiveBayesModelData(String[] featureNames, Map<Object, Double>[][] theta, double[] piArray, Object[] label) {
+ this.featureNames = featureNames;
+ this.theta = theta;
+ this.piArray = piArray;
+ this.label = label;
+ }
+
+ /** Encoder for the KMeans model data. */
+ public static class ModelDataEncoder implements Encoder<NaiveBayesModelData> {
+ @Override
+ public void encode(NaiveBayesModelData modelData, OutputStream outputStream) {
+ Kryo kryo = new Kryo();
+ Output output = new Output(outputStream);
+ kryo.writeObject(output, modelData);
+ output.flush();
+ }
+ }
+
+ /** Decoder for the KMeans model data. */
Review comment:
update the comment here.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
Review comment:
When we use `windowAll`, we are actually changing paralllelism to one and using one task to process all the elements.
Are there any efficiency issues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r757201463
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Table trainTable;
+ private Table predictTable;
+ private Map<Vector, Integer> expectedOutput;
+ private NaiveBayes estimator;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ Schema trainSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .column("f1", DataTypes.INT())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 0.), 11),
+ Row.of(Vectors.dense(1, 0), 10),
+ Row.of(Vectors.dense(1, 1.), 10))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ trainSchema)
+ .as("features", "label");
+
+ Schema predictSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 1.)),
+ Row.of(Vectors.dense(0, 0.)),
+ Row.of(Vectors.dense(1, 0)),
+ Row.of(Vectors.dense(1, 1.)))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ predictSchema)
+ .as("features");
+
+ expectedOutput =
+ new HashMap<Vector, Integer>() {
+ {
+ put(Vectors.dense(0, 1.), 11);
+ put(Vectors.dense(0, 0.), 11);
+ put(Vectors.dense(1, 0.), 10);
+ put(Vectors.dense(1, 1.), 10);
+ }
+ };
+
+ estimator =
+ new NaiveBayes()
+ .setSmoothing(1.0)
+ .setFeaturesCol("features")
+ .setLabelCol("label")
+ .setPredictionCol("predict")
+ .setModelType("multinomial");
+ }
+
+ /**
+ * Executes a given table and collect its results. Results are returned as a map whose key is
+ * the feature, value is the prediction result.
+ *
+ * @param table A table to be executed and to have its result collected
+ * @param featuresCol Name of the column in the table that contains the features
+ * @param predictionCol Name of the column in the table that contains the prediction result
+ * @return A map containing the collected results
+ */
+ private static Map<Vector, Integer> executeAndCollect(
+ Table table, String featuresCol, String predictionCol) {
+ Map<Vector, Integer> map = new HashMap<>();
+ for (CloseableIterator<Row> it = table.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ map.put(
+ (Vector) row.getField(featuresCol),
+ ((Number) row.getField(predictionCol)).intValue());
+ }
+
+ return map;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testFitAndPredict() throws Exception {
+ NaiveBayesModel model = estimator.fit(trainTable);
+ Table outputTable = model.transform(predictTable)[0];
+ Map<Vector, Integer> actualOutput =
+ executeAndCollect(outputTable, model.getFeaturesCol(), model.getPredictionCol());
+ assertEquals(expectedOutput, actualOutput);
+ }
+
+ @Test(expected = Exception.class)
+ public void testPredictUnseenFeature() throws Exception {
Review comment:
I think `testPredictUnseenFeature` should fail naturally while the program cannot find corresponding key of the feature. Adding a check and set detailed error message might have a bit influence on the performance of this algorithm. Other tests that expect an exception needs to check the message and I'll make this change on those tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r758021592
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The model data of {@link NaiveBayesModel}.
+ *
+ * <p>This class also provides methods to convert model data between Table and Datastream, and
+ * classes to save/load model data.
+ */
+public class NaiveBayesModelData implements Serializable {
+ public final Map<Double, Double>[][] theta;
+ public final double[] piArray;
+ public final int[] labels;
+
+ public NaiveBayesModelData(Map<Double, Double>[][] theta, double[] piArray, int[] labels) {
+ this.theta = theta;
+ this.piArray = piArray;
+ this.labels = labels;
+ }
+
+ /** Converts the provided modelData Datastream into corresponding Table. */
+ public static Table getModelDataTable(
+ StreamTableEnvironment tEnv, DataStream<NaiveBayesModelData> stream) {
Review comment:
Can we get `tEnv` from the input data stream rather than passing it as a param?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The model data of {@link NaiveBayesModel}.
+ *
+ * <p>This class also provides methods to convert model data between Table and Datastream, and
+ * classes to save/load model data.
+ */
+public class NaiveBayesModelData implements Serializable {
+ public final Map<Double, Double>[][] theta;
+ public final double[] piArray;
+ public final int[] labels;
+
+ public NaiveBayesModelData(Map<Double, Double>[][] theta, double[] piArray, int[] labels) {
+ this.theta = theta;
+ this.piArray = piArray;
+ this.labels = labels;
+ }
+
+ /** Converts the provided modelData Datastream into corresponding Table. */
+ public static Table getModelDataTable(
+ StreamTableEnvironment tEnv, DataStream<NaiveBayesModelData> stream) {
+ return tEnv.fromDataStream(stream);
+ }
+
+ /** Converts the provided modelData Table into corresponding DataStream. */
+ public static DataStream<NaiveBayesModelData> getModelDataStream(
+ StreamTableEnvironment tEnv, Table table) {
+ return tEnv.toDataStream(table)
Review comment:
same as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] gaoyunhaii commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r760901872
##########
File path: flink-ml-core/pom.xml
##########
@@ -65,5 +65,12 @@ under the License.
<artifactId>flink-shaded-jackson</artifactId>
<scope>provided</scope>
</dependency>
+
+ <dependency>
+ <groupId>dev.ludovic.netlib</groupId>
+ <artifactId>blas</artifactId>
+ <version>2.2.0</version>
+ <scope>compile</scope>
Review comment:
compile is the default scope, which might not need to be declared explicitly?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Integer>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Integer>>() {
+ @Override
+ public Tuple2<Vector, Integer> map(Row row) throws Exception {
+ Number number = (Number) row.getField(labelCol);
+ Preconditions.checkArgument(
+ number.intValue() == number.doubleValue());
Review comment:
Check if number is null?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The model data of {@link NaiveBayesModel}.
+ *
+ * <p>This class also provides methods to convert model data between Table and Datastream, and
+ * classes to save/load model data.
+ */
+public class NaiveBayesModelData implements Serializable {
+ public final Map<Double, Double>[][] theta;
Review comment:
Might add comments for the meanings of the fields ?
##########
File path: flink-ml-core/src/main/java/org/apache/flink/ml/linalg/BLAS.java
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.linalg;
+
+import org.apache.flink.util.Preconditions;
+
+/** A utility class that provides BLAS routines over matrices and vectors. */
+public class BLAS {
+ /** For level-1 routines, we use Java implementation. */
+ private static final dev.ludovic.netlib.BLAS F2J_BLAS =
Review comment:
What does `F2J_` mean?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Integer>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Integer>>() {
+ @Override
+ public Tuple2<Vector, Integer> map(Row row) throws Exception {
+ Number number = (Number) row.getField(labelCol);
+ Preconditions.checkArgument(
+ number.intValue() == number.doubleValue());
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ number.intValue());
+ }
+ });
+
+ DataStream<NaiveBayesModelData> modelData =
+ input.flatMap(new ExtractFeatureFunction())
+ .keyBy(
+ (KeySelector<Tuple4<Integer, Integer, Double, Double>, Object>)
Review comment:
Is the type casting necessary? Also for the following ones
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750876816
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
Review comment:
In latest commit I have replaced featureCols with featuresCol, from string array to a single string. So there is no need for duplicated check here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750848588
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
Review comment:
Following Spark's practice, I have changed input features into `Vector` and label into `double`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r756597745
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Object.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelStream =
+ NaiveBayesModelData.toDataStream(tEnv, modelTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(
+ new PredictLabelFunction(featuresCol, broadcastModelKey)));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.toDataStream(tEnv, modelTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(StreamExecutionEnvironment env, String path)
+ throws IOException {
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ Source<NaiveBayesModelData, ?, ?> source =
+ FileSource.forRecordStreamFormat(
+ new NaiveBayesModelData.ModelDataStreamFormat(),
+ ReadWriteUtils.getDataPaths(path))
+ .build();
+ NaiveBayesModel model = ReadWriteUtils.loadStageParam(path);
+ DataStream<NaiveBayesModelData> modelData =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "modelData");
+ model.setModelData(NaiveBayesModelData.fromDataStream(tEnv, modelData));
+
+ return model;
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ @Override
+ public NaiveBayesModel setModelData(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ modelTable = inputs[0];
+ return this;
+ }
+
+ @Override
+ public Table[] getModelData() {
+ return new Table[] {modelTable};
+ }
+
+ private static class PredictLabelOperator
+ extends AbstractUdfStreamOperator<Row, PredictLabelFunction>
+ implements OneInputStreamOperator<Row, Row> {
+ public PredictLabelOperator(PredictLabelFunction userFunction) {
+ super(userFunction);
+ }
+
+ @Override
+ public void processElement(StreamRecord<Row> streamRecord) {
Review comment:
You are right. This method is feasible. I have make this change and removed `PredictLabelFunction`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755670337
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/clustering/KMeansTest.java
##########
@@ -67,6 +69,18 @@
Vectors.dense(9.6, 0.0));
private StreamExecutionEnvironment env;
private StreamTableEnvironment tEnv;
+ private static List<Set<DenseVector>> expectedGroup =
Review comment:
OK. I'll fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755986602
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Object.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelStream =
+ NaiveBayesModelData.toDataStream(tEnv, modelTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(
+ new PredictLabelFunction(featuresCol, broadcastModelKey)));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.toDataStream(tEnv, modelTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(StreamExecutionEnvironment env, String path)
+ throws IOException {
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ Source<NaiveBayesModelData, ?, ?> source =
+ FileSource.forRecordStreamFormat(
+ new NaiveBayesModelData.ModelDataStreamFormat(),
+ ReadWriteUtils.getDataPaths(path))
+ .build();
+ NaiveBayesModel model = ReadWriteUtils.loadStageParam(path);
+ DataStream<NaiveBayesModelData> modelData =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "modelData");
+ model.setModelData(NaiveBayesModelData.fromDataStream(tEnv, modelData));
+
+ return model;
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ @Override
+ public NaiveBayesModel setModelData(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ modelTable = inputs[0];
+ return this;
+ }
+
+ @Override
+ public Table[] getModelData() {
+ return new Table[] {modelTable};
+ }
+
+ private static class PredictLabelOperator
+ extends AbstractUdfStreamOperator<Row, PredictLabelFunction>
+ implements OneInputStreamOperator<Row, Row> {
+ public PredictLabelOperator(PredictLabelFunction userFunction) {
+ super(userFunction);
+ }
+
+ @Override
+ public void processElement(StreamRecord<Row> streamRecord) {
Review comment:
Actually you can make the change if you use the RichMapFunction.
But I think making it a separate function is more clear to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r756556452
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Object.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelStream =
+ NaiveBayesModelData.toDataStream(tEnv, modelTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(
+ new PredictLabelFunction(featuresCol, broadcastModelKey)));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.toDataStream(tEnv, modelTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(StreamExecutionEnvironment env, String path)
+ throws IOException {
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ Source<NaiveBayesModelData, ?, ?> source =
+ FileSource.forRecordStreamFormat(
+ new NaiveBayesModelData.ModelDataStreamFormat(),
+ ReadWriteUtils.getDataPaths(path))
+ .build();
+ NaiveBayesModel model = ReadWriteUtils.loadStageParam(path);
+ DataStream<NaiveBayesModelData> modelData =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "modelData");
+ model.setModelData(NaiveBayesModelData.fromDataStream(tEnv, modelData));
+
+ return model;
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ @Override
+ public NaiveBayesModel setModelData(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ modelTable = inputs[0];
+ return this;
+ }
+
+ @Override
+ public Table[] getModelData() {
+ return new Table[] {modelTable};
+ }
+
+ private static class PredictLabelOperator
+ extends AbstractUdfStreamOperator<Row, PredictLabelFunction>
+ implements OneInputStreamOperator<Row, Row> {
+ public PredictLabelOperator(PredictLabelFunction userFunction) {
+ super(userFunction);
+ }
+
+ @Override
+ public void processElement(StreamRecord<Row> streamRecord) {
Review comment:
> I'm afraid that we cannot achieve this by now. The API withBroadcast requires that I must pass to it a child class of AbstractUdfStreamOperator. A child class of AbstractStreamOperator would throw exception.
Yes, the broadcast variables can only be accessed by `RichFunction#getRuntimeContext#getBroadcastVariable` now.
A possible solution to put processing logic in `PredictLabelOperator::processElement` is to add a AbstractFunction but only use it for accessing broadcast variables. An examples can be found in `org.apache.flink.ml.common.broadcast.operator.TestOneInputOp`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755986990
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Object.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelStream =
+ NaiveBayesModelData.toDataStream(tEnv, modelTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(
+ new PredictLabelFunction(featuresCol, broadcastModelKey)));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.toDataStream(tEnv, modelTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(StreamExecutionEnvironment env, String path)
+ throws IOException {
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ Source<NaiveBayesModelData, ?, ?> source =
+ FileSource.forRecordStreamFormat(
+ new NaiveBayesModelData.ModelDataStreamFormat(),
+ ReadWriteUtils.getDataPaths(path))
+ .build();
+ NaiveBayesModel model = ReadWriteUtils.loadStageParam(path);
+ DataStream<NaiveBayesModelData> modelData =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "modelData");
+ model.setModelData(NaiveBayesModelData.fromDataStream(tEnv, modelData));
+
+ return model;
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ @Override
+ public NaiveBayesModel setModelData(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ modelTable = inputs[0];
+ return this;
+ }
+
+ @Override
+ public Table[] getModelData() {
+ return new Table[] {modelTable};
+ }
+
+ private static class PredictLabelOperator
+ extends AbstractUdfStreamOperator<Row, PredictLabelFunction>
+ implements OneInputStreamOperator<Row, Row> {
+ public PredictLabelOperator(PredictLabelFunction userFunction) {
+ super(userFunction);
+ }
+
+ @Override
+ public void processElement(StreamRecord<Row> streamRecord) {
Review comment:
Actually you can make the change if you use the RichMapFunction.
But I think making it a separate function is more clear to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755986990
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Object.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelStream =
+ NaiveBayesModelData.toDataStream(tEnv, modelTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(
+ new PredictLabelFunction(featuresCol, broadcastModelKey)));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.toDataStream(tEnv, modelTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(StreamExecutionEnvironment env, String path)
+ throws IOException {
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ Source<NaiveBayesModelData, ?, ?> source =
+ FileSource.forRecordStreamFormat(
+ new NaiveBayesModelData.ModelDataStreamFormat(),
+ ReadWriteUtils.getDataPaths(path))
+ .build();
+ NaiveBayesModel model = ReadWriteUtils.loadStageParam(path);
+ DataStream<NaiveBayesModelData> modelData =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "modelData");
+ model.setModelData(NaiveBayesModelData.fromDataStream(tEnv, modelData));
+
+ return model;
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ @Override
+ public NaiveBayesModel setModelData(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ modelTable = inputs[0];
+ return this;
+ }
+
+ @Override
+ public Table[] getModelData() {
+ return new Table[] {modelTable};
+ }
+
+ private static class PredictLabelOperator
+ extends AbstractUdfStreamOperator<Row, PredictLabelFunction>
+ implements OneInputStreamOperator<Row, Row> {
+ public PredictLabelOperator(PredictLabelFunction userFunction) {
+ super(userFunction);
+ }
+
+ @Override
+ public void processElement(StreamRecord<Row> streamRecord) {
Review comment:
Actually you can make the change if you use the RichMapFunction.
But I think making it a separate function is more clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r757180944
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/util/StageTestUtils.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.util;
+
+import org.apache.flink.ml.api.core.Stage;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import org.junit.rules.TemporaryFolder;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+
+/** Utility methods for testing stages. */
+public class StageTestUtils {
+ private static final TemporaryFolder tempFolder = new TemporaryFolder();
+
+ static {
+ try {
+ tempFolder.create();
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ /**
+ * Saves a stage to filesystem and reloads it with the static load() method a stage must
Review comment:
Yes it's reasonable. I'll make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r751884914
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ final String featuresCol = getFeaturesCol();
Review comment:
OK. I'll add this check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r764652547
##########
File path: flink-ml-core/src/main/java/org/apache/flink/ml/linalg/BLAS.java
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.linalg;
+
+import org.apache.flink.util.Preconditions;
+
+/** A utility class that provides BLAS routines over matrices and vectors. */
+public class BLAS {
+ /** For level-1 routines, we use Java implementation. */
+ private static final dev.ludovic.netlib.BLAS F2J_BLAS =
Review comment:
It means that the implementation is translated from Fortran to Java. You can refer to Alink or Spark for the difference between NATIVE_BLAS and F2J_BLAS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755747666
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/util/StageTestUtils.java
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.util;
+
+import org.apache.flink.ml.api.core.Stage;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import java.lang.reflect.Method;
+import java.nio.file.Files;
+
+/** Utility methods for testing stages. */
+public class StageTestUtils {
+ /**
+ * Saves a stage to filesystem and reloads it with the static load() method a stage must
+ * implement.
+ */
+ public static <T extends Stage<T>> T saveAndReload(StreamExecutionEnvironment env, T stage)
+ throws Exception {
+ String tempDir = Files.createTempDirectory("").toString();
Review comment:
OK. I'll do it.
The common practice in Flink tests is to initialize the `TemporaryFolder` in each test class, and calls for initialization in `@Before` methods. Since the usage of `TemporaryFolder` in our case is merely in static utility functions. I think we can just put it as a static member of that class and initializes in static context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755726294
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+
+import static org.apache.flink.table.api.Expressions.$;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private List<Row> trainData;
+ private List<Row> predictData;
+ private List<Row> expectedOutput;
+ private boolean isSaveLoad;
Review comment:
Yes you are right. I will make this change in `NaiveBayesTest` like what we have done in `KmeansTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r762804604
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The model data of {@link NaiveBayesModel}.
+ *
+ * <p>This class also provides methods to convert model data between Table and Datastream, and
+ * classes to save/load model data.
+ */
+public class NaiveBayesModelData implements Serializable {
Review comment:
can we remove `implements Serializable` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r749928870
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
Review comment:
There could be efficiency issues, but I think this is a necessity, because we need one operator to gather all the partial changes to form a `NaiveBayesModelData` before broadcasting it to `Model` operators. Before it is possible to avoid reducing parallelism to one, we need a way to always calculate and store a model data in a separated form.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750846697
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesParams.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.ml.param.DoubleParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.shared.colname.HasFeatureCols;
+import org.apache.flink.ml.param.shared.colname.HasLabelCol;
+import org.apache.flink.ml.param.shared.colname.HasPredictionCol;
+
+/**
+ * Parameters of naive bayes training process.
+ */
+public interface NaiveBayesParams<T> extends
+ HasFeatureCols<T>,
+ HasLabelCol<T>,
+ HasPredictionCol<T> {
+ Param<Double> SMOOTHING =
+ new DoubleParam(
+ "smoothing",
+ "the smoothing factor",
Review comment:
OK I'll update the description. I use "smoothing factor" because Alink describes in this way. I'll update it into Spark's convention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] gaoyunhaii closed pull request #32: [FLINK-24817] Add Estimator and Transformer for Naive Bayes
Posted by GitBox <gi...@apache.org>.
gaoyunhaii closed pull request #32:
URL: https://github.com/apache/flink-ml/pull/32
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r765390546
##########
File path: flink-ml-core/src/main/java/org/apache/flink/ml/linalg/BLAS.java
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.linalg;
+
+import org.apache.flink.util.Preconditions;
+
+/** A utility class that provides BLAS routines over matrices and vectors. */
+public class BLAS {
+ /** For level-1 routines, we use Java implementation. */
+ private static final dev.ludovic.netlib.BLAS F2J_BLAS =
Review comment:
Oh sorry. I also found that `F2J_BLAS` is only available in Alink and in `com.github.fommil.netlib`. I'll change the comment and naming to Spark's style.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r764666613
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The model data of {@link NaiveBayesModel}.
+ *
+ * <p>This class also provides methods to convert model data between Table and Datastream, and
+ * classes to save/load model data.
+ */
+public class NaiveBayesModelData implements Serializable {
Review comment:
Yes. I'll do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r764623110
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The model data of {@link NaiveBayesModel}.
+ *
+ * <p>This class also provides methods to convert model data between Table and Datastream, and
+ * classes to save/load model data.
+ */
+public class NaiveBayesModelData implements Serializable {
+ public final Map<Double, Double>[][] theta;
+ public final double[] piArray;
+ public final int[] labels;
+
+ public NaiveBayesModelData(Map<Double, Double>[][] theta, double[] piArray, int[] labels) {
+ this.theta = theta;
+ this.piArray = piArray;
+ this.labels = labels;
+ }
+
+ /** Converts the provided modelData Datastream into corresponding Table. */
+ public static Table getModelDataTable(DataStream<NaiveBayesModelData> stream) {
+ StreamTableEnvironment tEnv =
+ StreamTableEnvironment.create(stream.getExecutionEnvironment());
+ return tEnv.fromDataStream(stream);
+ }
+
+ /** Converts the provided modelData Table into corresponding DataStream. */
+ public static DataStream<NaiveBayesModelData> getModelDataStream(Table table) {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) table).getTableEnvironment();
+ return tEnv.toDataStream(table)
+ .map(
+ (MapFunction<Row, NaiveBayesModelData>)
+ row -> (NaiveBayesModelData) row.getField("f0"));
+ }
+
+ /** Encoder for the {@link NaiveBayesModelData}. */
+ public static class ModelDataEncoder implements Encoder<NaiveBayesModelData> {
+ @Override
+ public void encode(NaiveBayesModelData modelData, OutputStream outputStream) {
+ Output output = new Output(outputStream);
+
+ output.writeInt(modelData.labels.length);
Review comment:
OK. I'll make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r748081703
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
+ return;
+ }
+
+ for (int i = 0; i < featureSize; i++) {
+ Object feature = row.getField(featureColNames[i]);
+ if (feature == null) {
+ continue;
+ }
+ collector.collect(new Tuple4<>(label, i, feature, 1.0));
Review comment:
Shall we support weighted naive bayes or just remove 1.0 here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750848317
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
+ return;
+ }
+
+ for (int i = 0; i < featureSize; i++) {
+ Object feature = row.getField(featureColNames[i]);
+ if (feature == null) {
+ continue;
+ }
+ collector.collect(new Tuple4<>(label, i, feature, 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and feature column index.
+ */
+ private static class ValueMapFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Object, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>> {
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> createAccumulator() {
+ return new Tuple4<>(new Object(), -1, new HashMap<>(), 0.);
Review comment:
OK. I have removed the field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750846941
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
Review comment:
Got it. I have removed redundant checks and added Flink's `Precondition` checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750845569
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.ObjectArrayTypeInfo;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.Map;
+
+/**
+ * The model data of {@link NaiveBayesModel}.
+ */
+public class NaiveBayesModelData implements Serializable {
+ private static final long serialVersionUID = 3919917903722286395L;
+ public final String[] featureNames;
+ public final Map<Object, Double>[][] theta;
+ public final double[] piArray;
+ public final Object[] label;
+
+ public static final Schema SCHEMA =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(NaiveBayesModelData.class))
+ .build();
+
+ public NaiveBayesModelData(String[] featureNames, Map<Object, Double>[][] theta, double[] piArray, Object[] label) {
+ this.featureNames = featureNames;
+ this.theta = theta;
+ this.piArray = piArray;
+ this.label = label;
+ }
+
+ /** Encoder for the KMeans model data. */
+ public static class ModelDataEncoder implements Encoder<NaiveBayesModelData> {
+ @Override
+ public void encode(NaiveBayesModelData modelData, OutputStream outputStream) {
+ Kryo kryo = new Kryo();
+ Output output = new Output(outputStream);
+ kryo.writeObject(output, modelData);
+ output.flush();
+ }
+ }
+
+ /** Decoder for the KMeans model data. */
Review comment:
OK. I have fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r749920020
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
+ return;
+ }
+
+ for (int i = 0; i < featureSize; i++) {
+ Object feature = row.getField(featureColNames[i]);
+ if (feature == null) {
+ continue;
+ }
+ collector.collect(new Tuple4<>(label, i, feature, 1.0));
Review comment:
We may be able to support weighted training data in Naive Bayes in a later version, but in the current version I believe we only need to achieve the basic functionality of this algorithm.
Although we do not need weighted function here in this version, we still need to add this `1.0` to every training data, because we need to sum up this value and use `weightSum` to calculate the probability of each feature value and each label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755759161
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Double>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Double>>() {
+ @Override
+ public Tuple2<Vector, Double> map(Row row) throws Exception {
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ (Double) row.getField(labelCol));
+ }
+ });
+
+ DataStream<NaiveBayesModelData> naiveBayesModel =
+ input.flatMap(new FlattenFunction())
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce(
+ (ReduceFunction<Tuple4<Double, Integer, Double, Double>>)
+ (t0, t1) -> {
+ t0.f3 += t1.f3;
+ return t0;
+ })
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy(
+ (KeySelector<
+ Tuple4<
+ Double,
+ Integer,
+ Map<Double, Double>,
+ Double>,
+ Object>)
+ value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction())
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(smoothing));
+
+ NaiveBayesModel model =
+ new NaiveBayesModel()
+ .setModelData(NaiveBayesModelData.fromDataStream(tEnv, naiveBayesModel));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(StreamExecutionEnvironment env, String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value,
+ * weight.
+ */
+ private static class FlattenFunction
+ implements FlatMapFunction<
+ Tuple2<Vector, Double>, Tuple4<Double, Integer, Double, Double>> {
Review comment:
Yes, that's right. I'll make the change.
I'll make it that users can still choose to input labels that are integer or double, but finally they will be converted to integer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r757201463
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Table trainTable;
+ private Table predictTable;
+ private Map<Vector, Integer> expectedOutput;
+ private NaiveBayes estimator;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ Schema trainSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .column("f1", DataTypes.INT())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 0.), 11),
+ Row.of(Vectors.dense(1, 0), 10),
+ Row.of(Vectors.dense(1, 1.), 10))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ trainSchema)
+ .as("features", "label");
+
+ Schema predictSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 1.)),
+ Row.of(Vectors.dense(0, 0.)),
+ Row.of(Vectors.dense(1, 0)),
+ Row.of(Vectors.dense(1, 1.)))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ predictSchema)
+ .as("features");
+
+ expectedOutput =
+ new HashMap<Vector, Integer>() {
+ {
+ put(Vectors.dense(0, 1.), 11);
+ put(Vectors.dense(0, 0.), 11);
+ put(Vectors.dense(1, 0.), 10);
+ put(Vectors.dense(1, 1.), 10);
+ }
+ };
+
+ estimator =
+ new NaiveBayes()
+ .setSmoothing(1.0)
+ .setFeaturesCol("features")
+ .setLabelCol("label")
+ .setPredictionCol("predict")
+ .setModelType("multinomial");
+ }
+
+ /**
+ * Executes a given table and collect its results. Results are returned as a map whose key is
+ * the feature, value is the prediction result.
+ *
+ * @param table A table to be executed and to have its result collected
+ * @param featuresCol Name of the column in the table that contains the features
+ * @param predictionCol Name of the column in the table that contains the prediction result
+ * @return A map containing the collected results
+ */
+ private static Map<Vector, Integer> executeAndCollect(
+ Table table, String featuresCol, String predictionCol) {
+ Map<Vector, Integer> map = new HashMap<>();
+ for (CloseableIterator<Row> it = table.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ map.put(
+ (Vector) row.getField(featuresCol),
+ ((Number) row.getField(predictionCol)).intValue());
+ }
+
+ return map;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testFitAndPredict() throws Exception {
+ NaiveBayesModel model = estimator.fit(trainTable);
+ Table outputTable = model.transform(predictTable)[0];
+ Map<Vector, Integer> actualOutput =
+ executeAndCollect(outputTable, model.getFeaturesCol(), model.getPredictionCol());
+ assertEquals(expectedOutput, actualOutput);
+ }
+
+ @Test(expected = Exception.class)
+ public void testPredictUnseenFeature() throws Exception {
Review comment:
I think We can check the expection class and stack trace, but not the error message. I think that`testPredictUnseenFeature` should fail naturally while the program cannot find corresponding value of the feature. Adding an additional check and set detailed error message might have a bit influence on the performance of this algorithm. Other tests that expect an exception do need to check the message and I'll make this change on those tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755766209
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Object.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelStream =
+ NaiveBayesModelData.toDataStream(tEnv, modelTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(
+ new PredictLabelFunction(featuresCol, broadcastModelKey)));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.toDataStream(tEnv, modelTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(StreamExecutionEnvironment env, String path)
+ throws IOException {
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ Source<NaiveBayesModelData, ?, ?> source =
+ FileSource.forRecordStreamFormat(
+ new NaiveBayesModelData.ModelDataStreamFormat(),
+ ReadWriteUtils.getDataPaths(path))
+ .build();
+ NaiveBayesModel model = ReadWriteUtils.loadStageParam(path);
+ DataStream<NaiveBayesModelData> modelData =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "modelData");
+ model.setModelData(NaiveBayesModelData.fromDataStream(tEnv, modelData));
+
+ return model;
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ @Override
+ public NaiveBayesModel setModelData(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ modelTable = inputs[0];
+ return this;
+ }
+
+ @Override
+ public Table[] getModelData() {
+ return new Table[] {modelTable};
+ }
+
+ private static class PredictLabelOperator
+ extends AbstractUdfStreamOperator<Row, PredictLabelFunction>
+ implements OneInputStreamOperator<Row, Row> {
+ public PredictLabelOperator(PredictLabelFunction userFunction) {
+ super(userFunction);
+ }
+
+ @Override
+ public void processElement(StreamRecord<Row> streamRecord) {
Review comment:
I'm afraid that we cannot achieve this by now. The API `withBroadcast` requires that I must pass to it a child class of `AbstractUdfStreamOperator`. A child class of `AbstractStreamOperator` would throw exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
zhipeng93 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r756556452
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Object.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelStream =
+ NaiveBayesModelData.toDataStream(tEnv, modelTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(
+ new PredictLabelFunction(featuresCol, broadcastModelKey)));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.toDataStream(tEnv, modelTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(StreamExecutionEnvironment env, String path)
+ throws IOException {
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ Source<NaiveBayesModelData, ?, ?> source =
+ FileSource.forRecordStreamFormat(
+ new NaiveBayesModelData.ModelDataStreamFormat(),
+ ReadWriteUtils.getDataPaths(path))
+ .build();
+ NaiveBayesModel model = ReadWriteUtils.loadStageParam(path);
+ DataStream<NaiveBayesModelData> modelData =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "modelData");
+ model.setModelData(NaiveBayesModelData.fromDataStream(tEnv, modelData));
+
+ return model;
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ @Override
+ public NaiveBayesModel setModelData(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ modelTable = inputs[0];
+ return this;
+ }
+
+ @Override
+ public Table[] getModelData() {
+ return new Table[] {modelTable};
+ }
+
+ private static class PredictLabelOperator
+ extends AbstractUdfStreamOperator<Row, PredictLabelFunction>
+ implements OneInputStreamOperator<Row, Row> {
+ public PredictLabelOperator(PredictLabelFunction userFunction) {
+ super(userFunction);
+ }
+
+ @Override
+ public void processElement(StreamRecord<Row> streamRecord) {
Review comment:
>
Ah, I see.
Yes, the broadcast variables can only be accessed by `RichFunction#getRuntimeContext#getBroadcastVariable` now.
A possible solution to put processing logic in `PredictLabelOperator::processElement` is to add a AbstractFunction but only use it for accessing broadcast variables. An examples can be found in `org.apache.flink.ml.common.broadcast.operator.TestOneInputOp`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r757181326
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Integer>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Integer>>() {
+ @Override
+ public Tuple2<Vector, Integer> map(Row row) throws Exception {
+ Number number = (Number) row.getField(labelCol);
+ Preconditions.checkArgument(
+ number.intValue() == number.doubleValue());
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ number.intValue());
+ }
+ });
+
+ DataStream<NaiveBayesModelData> modelData =
+ input.flatMap(new ExtractFeatureFunction())
+ .keyBy(
+ (KeySelector<Tuple4<Integer, Integer, Double, Double>, Object>)
+ value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce(
+ (ReduceFunction<Tuple4<Integer, Integer, Double, Double>>)
+ (t0, t1) ->
+ new Tuple4<>(t0.f0, t0.f1, t0.f2, t0.f3 + t1.f3))
+ .keyBy(
+ (KeySelector<Tuple4<Integer, Integer, Double, Double>, Object>)
+ value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new GenerateFeatureWeightMapFunction())
+ .keyBy(
+ (KeySelector<
+ Tuple4<
+ Integer,
+ Integer,
+ Map<Double, Double>,
+ Double>,
+ Object>)
+ value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new AggregateIntoArrayFunction())
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(smoothing));
+
+ NaiveBayesModel model =
+ new NaiveBayesModel()
+ .setModelData(NaiveBayesModelData.getModelDataTable(tEnv, modelData));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(StreamExecutionEnvironment env, String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to extract feature values from input rows.
+ *
+ * <p>Output records are tuples with the following fields in order:
+ *
+ * <ul>
+ * <li>label value
+ * <li>feature column index
+ * <li>feature value
+ * <li>weight
+ * </ul>
+ */
+ private static class ExtractFeatureFunction
+ implements FlatMapFunction<
+ Tuple2<Vector, Integer>, Tuple4<Integer, Integer, Double, Double>> {
+ @Override
+ public void flatMap(
+ Tuple2<Vector, Integer> value,
+ Collector<Tuple4<Integer, Integer, Double, Double>> collector) {
+ Preconditions.checkNotNull(value.f1);
+ for (int i = 0; i < value.f0.size(); i++) {
+ collector.collect(new Tuple4<>(value.f1, i, value.f0.get(i), 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function that aggregates entries of feature value and weight into maps.
+ *
+ * <p>Input records should have the same label value and feature column index.
+ *
+ * <p>Input records are tuples with the following fields in order:
+ *
+ * <ul>
+ * <li>label value
+ * <li>feature column index
+ * <li>feature value
+ * <li>weight
+ * </ul>
+ *
+ * <p>Output records are tuples with the following fields in order:
+ *
+ * <ul>
+ * <li>label value
+ * <li>feature column index
+ * <li>map of (feature value, weight)
+ * </ul>
+ */
+ private static class GenerateFeatureWeightMapFunction
+ implements AggregateFunction<
+ Tuple4<Integer, Integer, Double, Double>,
+ Tuple3<Integer, Integer, Map<Double, Double>>,
+ Tuple4<Integer, Integer, Map<Double, Double>, Double>> {
+
+ @Override
+ public Tuple3<Integer, Integer, Map<Double, Double>> createAccumulator() {
+ return new Tuple3<>(0, -1, new HashMap<>());
+ }
+
+ @Override
+ public Tuple3<Integer, Integer, Map<Double, Double>> add(
+ Tuple4<Integer, Integer, Double, Double> value,
+ Tuple3<Integer, Integer, Map<Double, Double>> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Integer, Integer, Map<Double, Double>, Double> getResult(
+ Tuple3<Integer, Integer, Map<Double, Double>> acc) {
+ double weightSum = acc.f2.values().stream().mapToDouble(Double::doubleValue).sum();
+ return new Tuple4<>(acc.f0, acc.f1, acc.f2, weightSum);
+ }
+
+ @Override
+ public Tuple3<Integer, Integer, Map<Double, Double>> merge(
+ Tuple3<Integer, Integer, Map<Double, Double>> acc0,
+ Tuple3<Integer, Integer, Map<Double, Double>> acc1) {
+ Preconditions.checkArgument(acc0.f1 != -1);
+ acc0.f2.putAll(acc1.f2);
+ return acc0;
+ }
+ }
+
+ /**
+ * Function that aggregates maps under the same label into arrays.
+ *
+ * <p>Length of the generated array equals to the number of feature columns.
+ *
+ * <p>Input records are tuples with the following fields in order:
+ *
+ * <ul>
+ * <li>label value
+ * <li>feature column index
+ * <li>map of (feature value, weight)
Review comment:
Yes. I'll fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750847518
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
Review comment:
OK. I have added `ParamValidators::notNull` to `HasLabelCol` and calls `getLabelCol()` once `NaiveBayes.fit()` is called in order to invoke the validator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r749965528
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
Review comment:
I agree we can remove this check.
My consideration was that, like we want to throw NPE instead of ignoring null values or labels, duplicated feature column also faces such choice. We could explicitly throw an exception and let users handle it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750847956
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
+ return;
+ }
+
+ for (int i = 0; i < featureSize; i++) {
+ Object feature = row.getField(featureColNames[i]);
+ if (feature == null) {
+ continue;
+ }
+ collector.collect(new Tuple4<>(label, i, feature, 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and feature column index.
+ */
+ private static class ValueMapFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Object, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>> {
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> createAccumulator() {
+ return new Tuple4<>(new Object(), -1, new HashMap<>(), 0.);
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> add(
+ Tuple4<Object, Integer, Object, Double> value,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ acc.f3 += value.f3;
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> getResult(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> merge(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc0,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc1) {
+ if (acc0.f0.equals(new Object())) {
+ acc0.f0 = acc1.f0;
+ }
+ if (acc0.f1 == -1) {
+ acc0.f1 = acc1.f1;
+ }
+ for (Map.Entry<Object, Double> entry: acc1.f2.entrySet()) {
+ acc0.f2.put(entry.getKey(), entry.getValue());
+ }
+ acc0.f3 += acc1.f3;
+
+ return acc0;
+ }
+ }
+
+ /**
+ * Function to aggregate maps under the same label into arrays. array len = featureSize
+ */
+ private static class MapArrayFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple3<Object, Double[], Map <Object, Double>[]>,
+ Tuple3<Object, Double[], Map <Object, Double>[]>> {
+ private final int featureSize;
+
+ private MapArrayFunction(int featureSize) {
+ this.featureSize = featureSize;
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> createAccumulator() {
+ Double[] weightSum = new Double[featureSize];
+ Arrays.fill(weightSum, 0.);
+ return new Tuple3<>(new Object(), weightSum, new HashMap[featureSize]);
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> add(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> value,
+ Tuple3<Object, Double[], Map<Object, Double>[]> acc) {
+ acc.f0 = value.f0;
+ acc.f1[value.f1] = value.f3;
+ acc.f2[value.f1] = value.f2;
+
+ return acc;
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> getResult(Tuple3<Object, Double[], Map<Object, Double>[]> acc) {
+ return acc;
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> merge(
+ Tuple3<Object, Double[], Map<Object, Double>[]> acc0,
+ Tuple3<Object, Double[], Map<Object, Double>[]> acc1) {
+ if (acc0.f0.equals(new Object())) {
+ acc0.f0 = acc1.f0;
+ }
+ for (int i = 0; i < featureSize; i++) {
+ acc0.f1[i] += acc1.f1[i];
+ if (acc1.f2[i] != null) {
Review comment:
We need to merge two arrays whose elements are not initialized. Thus it is possible to contain null values in this array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
lindong28 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r749901463
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesParams.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.ml.param.DoubleParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.shared.colname.HasFeatureCols;
+import org.apache.flink.ml.param.shared.colname.HasLabelCol;
+import org.apache.flink.ml.param.shared.colname.HasPredictionCol;
+
+/**
+ * Parameters of naive bayes training process.
+ */
+public interface NaiveBayesParams<T> extends
Review comment:
Could we add a parameter like modelType here? I suppose we will eventually expand NaiveBayes to support all modelTypes currently supported by Spark.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesParams.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.ml.param.DoubleParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.shared.colname.HasFeatureCols;
+import org.apache.flink.ml.param.shared.colname.HasLabelCol;
+import org.apache.flink.ml.param.shared.colname.HasPredictionCol;
+
+/**
+ * Parameters of naive bayes training process.
+ */
+public interface NaiveBayesParams<T> extends
+ HasFeatureCols<T>,
+ HasLabelCol<T>,
+ HasPredictionCol<T> {
+ Param<Double> SMOOTHING =
+ new DoubleParam(
+ "smoothing",
+ "the smoothing factor",
+ 0.0,
Review comment:
The default value in Spark is 1.0. Do you think 1.0 would be a better default value here?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesParams.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.ml.param.DoubleParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.shared.colname.HasFeatureCols;
+import org.apache.flink.ml.param.shared.colname.HasLabelCol;
+import org.apache.flink.ml.param.shared.colname.HasPredictionCol;
+
+/**
+ * Parameters of naive bayes training process.
+ */
+public interface NaiveBayesParams<T> extends
+ HasFeatureCols<T>,
+ HasLabelCol<T>,
+ HasPredictionCol<T> {
+ Param<Double> SMOOTHING =
+ new DoubleParam(
+ "smoothing",
+ "the smoothing factor",
Review comment:
How about we unanimously start the parameter doc with an upper-case character and end it with `.`, e.g. `The smoothing factor.`. Same for other parameters. I will update the KMeans PR to follow the same convention.
Another minor question is, would `smoothing parameter` be better here? If I search `naive bayes smoothing` on Google, the next suggested word is `parameter` instead of `factor`. Spark also uses `parameter` in this case.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
Review comment:
It seems a bit verbose to explicitly check whether input is null in every function. It is OK to just throw NPE in this case. Can we remove this check?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
+ return;
+ }
+
+ for (int i = 0; i < featureSize; i++) {
+ Object feature = row.getField(featureColNames[i]);
+ if (feature == null) {
+ continue;
+ }
+ collector.collect(new Tuple4<>(label, i, feature, 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and feature column index.
+ */
+ private static class ValueMapFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Object, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>> {
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> createAccumulator() {
+ return new Tuple4<>(new Object(), -1, new HashMap<>(), 0.);
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> add(
+ Tuple4<Object, Integer, Object, Double> value,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ acc.f3 += value.f3;
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> getResult(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> merge(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc0,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc1) {
+ if (acc0.f0.equals(new Object())) {
+ acc0.f0 = acc1.f0;
+ }
+ if (acc0.f1 == -1) {
+ acc0.f1 = acc1.f1;
+ }
+ for (Map.Entry<Object, Double> entry: acc1.f2.entrySet()) {
+ acc0.f2.put(entry.getKey(), entry.getValue());
+ }
+ acc0.f3 += acc1.f3;
+
+ return acc0;
+ }
+ }
+
+ /**
+ * Function to aggregate maps under the same label into arrays. array len = featureSize
+ */
+ private static class MapArrayFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple3<Object, Double[], Map <Object, Double>[]>,
+ Tuple3<Object, Double[], Map <Object, Double>[]>> {
+ private final int featureSize;
+
+ private MapArrayFunction(int featureSize) {
+ this.featureSize = featureSize;
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> createAccumulator() {
+ Double[] weightSum = new Double[featureSize];
+ Arrays.fill(weightSum, 0.);
+ return new Tuple3<>(new Object(), weightSum, new HashMap[featureSize]);
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> add(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> value,
+ Tuple3<Object, Double[], Map<Object, Double>[]> acc) {
+ acc.f0 = value.f0;
+ acc.f1[value.f1] = value.f3;
+ acc.f2[value.f1] = value.f2;
+
+ return acc;
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> getResult(Tuple3<Object, Double[], Map<Object, Double>[]> acc) {
+ return acc;
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> merge(
+ Tuple3<Object, Double[], Map<Object, Double>[]> acc0,
+ Tuple3<Object, Double[], Map<Object, Double>[]> acc1) {
+ if (acc0.f0.equals(new Object())) {
+ acc0.f0 = acc1.f0;
+ }
+ for (int i = 0; i < featureSize; i++) {
+ acc0.f1[i] += acc1.f1[i];
+ if (acc1.f2[i] != null) {
Review comment:
Would this ever happen?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
+ return;
+ }
+
+ for (int i = 0; i < featureSize; i++) {
+ Object feature = row.getField(featureColNames[i]);
+ if (feature == null) {
+ continue;
+ }
+ collector.collect(new Tuple4<>(label, i, feature, 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and feature column index.
+ */
+ private static class ValueMapFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Object, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>> {
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> createAccumulator() {
+ return new Tuple4<>(new Object(), -1, new HashMap<>(), 0.);
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> add(
+ Tuple4<Object, Integer, Object, Double> value,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ acc.f3 += value.f3;
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> getResult(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> merge(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc0,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc1) {
+ if (acc0.f0.equals(new Object())) {
+ acc0.f0 = acc1.f0;
+ }
+ if (acc0.f1 == -1) {
+ acc0.f1 = acc1.f1;
+ }
+ for (Map.Entry<Object, Double> entry: acc1.f2.entrySet()) {
Review comment:
It seems simpler to do `acc0.f2.putAll(acc1.f2)`?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
+ return;
+ }
+
+ for (int i = 0; i < featureSize; i++) {
+ Object feature = row.getField(featureColNames[i]);
+ if (feature == null) {
+ continue;
+ }
+ collector.collect(new Tuple4<>(label, i, feature, 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and feature column index.
+ */
+ private static class ValueMapFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Object, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>> {
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> createAccumulator() {
+ return new Tuple4<>(new Object(), -1, new HashMap<>(), 0.);
Review comment:
It seems that the last field (i.e. weight) can be removed since this information can be derived from the 3rd field (i.e. the map from feature value to weight).
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
Review comment:
Supporting arbitrary object as label/feature seems a bit too wide. What is the practice followed by Spark?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
Review comment:
This could be checked by using `ParamValidators::notNull` for the labelColumn parameter?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
+ return;
+ }
+
+ for (int i = 0; i < featureSize; i++) {
+ Object feature = row.getField(featureColNames[i]);
+ if (feature == null) {
+ continue;
+ }
+ collector.collect(new Tuple4<>(label, i, feature, 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and feature column index.
+ */
+ private static class ValueMapFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Object, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>> {
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> createAccumulator() {
+ return new Tuple4<>(new Object(), -1, new HashMap<>(), 0.);
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> add(
+ Tuple4<Object, Integer, Object, Double> value,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ acc.f3 += value.f3;
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> getResult(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> merge(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc0,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc1) {
+ if (acc0.f0.equals(new Object())) {
Review comment:
Would it be simpler to also do `acc0.f1 = acc1.f1` here?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
Review comment:
Should we throw exception here if `label` is not found in the data?
Sam for `feature`.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
Review comment:
Why do we need to explicitly check this? If we do need this, could this be enforced as a validator in the parameter definition?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ String[] featureColNames = getFeatureCols();
+ String labelColName = getLabelCol();
+ String predictionCol = getPredictionCol();
+ double smoothing = getSmoothing();
+
+ Preconditions.checkNotNull(inputs, "input table list should not be null");
+ Preconditions.checkArgument(inputs.length == 1, "input table list should contain only one argument");
+ Preconditions.checkArgument(
+ new HashSet<>(Arrays.asList(featureColNames)).size() == featureColNames.length,
+ "feature columns should not duplicate");
+ Preconditions.checkNotNull(labelColName, "label column should be set");
+
+ StreamTableEnvironment tEnv = (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ DataStream<NaiveBayesModelData> naiveBayesModel = input
+ .flatMap(new FlattenFunction(
+ featureColNames,
+ labelColName
+ ))
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce((ReduceFunction<Tuple4<Object, Integer, Object, Double>>) (t0, t1) -> {t0.f3 += t1.f3; return t0; })
+ .keyBy((KeySelector<Tuple4<Object, Integer, Object, Double>, Object>) value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy((KeySelector<Tuple4<Object, Integer, Map<Object, Double>, Double>, Object>) value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction(featureColNames.length))
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(
+ smoothing,
+ featureColNames));
+
+ NaiveBayesModel model = new NaiveBayesModel()
+ .setPredictionCol(predictionCol)
+ .setFeatureCols(featureColNames);
+ model.setModelData(
+ tEnv.fromDataStream(naiveBayesModel)
+ );
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value, weight.
+ */
+ private static class FlattenFunction implements FlatMapFunction<Row, Tuple4<Object, Integer, Object, Double>> {
+ private final String[] featureColNames;
+ private final String labelColName;
+ private final int featureSize;
+
+ private FlattenFunction(String[] featureColNames, String labelColName) {
+ this.labelColName = labelColName;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void flatMap(Row row, Collector<Tuple4<Object, Integer, Object, Double>> collector) {
+ Object label = row.getField(labelColName);
+ if (label == null) {
+ return;
+ }
+
+ for (int i = 0; i < featureSize; i++) {
+ Object feature = row.getField(featureColNames[i]);
+ if (feature == null) {
+ continue;
+ }
+ collector.collect(new Tuple4<>(label, i, feature, 1.0));
+ }
+ }
+ }
+
+ /**
+ * Function to aggregate feature value and weight into map from records with the same label and feature column index.
+ */
+ private static class ValueMapFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Object, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple4<Object, Integer, Map<Object, Double>, Double>> {
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> createAccumulator() {
+ return new Tuple4<>(new Object(), -1, new HashMap<>(), 0.);
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> add(
+ Tuple4<Object, Integer, Object, Double> value,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ acc.f0 = value.f0;
+ acc.f1 = value.f1;
+ acc.f2.put(value.f2, value.f3);
+ acc.f3 += value.f3;
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> getResult(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc) {
+ return acc;
+ }
+
+ @Override
+ public Tuple4<Object, Integer, Map<Object, Double>, Double> merge(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc0,
+ Tuple4<Object, Integer, Map<Object, Double>, Double> acc1) {
+ if (acc0.f0.equals(new Object())) {
+ acc0.f0 = acc1.f0;
+ }
+ if (acc0.f1 == -1) {
+ acc0.f1 = acc1.f1;
+ }
+ for (Map.Entry<Object, Double> entry: acc1.f2.entrySet()) {
+ acc0.f2.put(entry.getKey(), entry.getValue());
+ }
+ acc0.f3 += acc1.f3;
+
+ return acc0;
+ }
+ }
+
+ /**
+ * Function to aggregate maps under the same label into arrays. array len = featureSize
+ */
+ private static class MapArrayFunction implements AggregateFunction<
+ Tuple4<Object, Integer, Map<Object, Double>, Double>,
+ Tuple3<Object, Double[], Map <Object, Double>[]>,
+ Tuple3<Object, Double[], Map <Object, Double>[]>> {
+ private final int featureSize;
+
+ private MapArrayFunction(int featureSize) {
+ this.featureSize = featureSize;
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> createAccumulator() {
+ Double[] weightSum = new Double[featureSize];
+ Arrays.fill(weightSum, 0.);
+ return new Tuple3<>(new Object(), weightSum, new HashMap[featureSize]);
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> add(
+ Tuple4<Object, Integer, Map<Object, Double>, Double> value,
+ Tuple3<Object, Double[], Map<Object, Double>[]> acc) {
+ acc.f0 = value.f0;
+ acc.f1[value.f1] = value.f3;
+ acc.f2[value.f1] = value.f2;
+
+ return acc;
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> getResult(Tuple3<Object, Double[], Map<Object, Double>[]> acc) {
+ return acc;
+ }
+
+ @Override
+ public Tuple3<Object, Double[], Map<Object, Double>[]> merge(
+ Tuple3<Object, Double[], Map<Object, Double>[]> acc0,
+ Tuple3<Object, Double[], Map<Object, Double>[]> acc1) {
+ if (acc0.f0.equals(new Object())) {
+ acc0.f0 = acc1.f0;
+ }
+ for (int i = 0; i < featureSize; i++) {
+ acc0.f1[i] += acc1.f1[i];
+ if (acc1.f2[i] != null) {
+ acc0.f2[i] = acc1.f2[i];
+ }
+ }
+
+ return acc0;
+ }
+ }
+
+ /**
+ * Function to generate Naive Bayes model data.
+ */
+ private static class GenerateModelFunction implements AllWindowFunction<
+ Tuple3<Object, Double[], Map<Object, Double>[]>,
+ NaiveBayesModelData,
+ TimeWindow
+ > {
+ private final int featureSize;
+ private final double smoothing;
+ private final String[] featureColNames;
+
+ GenerateModelFunction(double smoothing, String[] featureColNames) {
+ this.smoothing = smoothing;
+ this.featureColNames = featureColNames;
+ this.featureSize = featureColNames.length;
+ }
+
+ @Override
+ public void apply(TimeWindow timeWindow,
+ Iterable<Tuple3<Object, Double[], Map<Object, Double>[]>> values,
+ Collector<NaiveBayesModelData> out) {
+ double[] numDocs = new double[featureSize];
+ ArrayList <Tuple3 <Object, Double[], Map <Object, Double>[]>> modelArray = new ArrayList <>();
+ HashSet <Object>[] categoryNumbers = new HashSet[featureSize];
Review comment:
Can you remove the dummy space here? Same for other code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
lindong28 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r751875906
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ final String featuresCol = getFeaturesCol();
Review comment:
I realized that it is probably better to setup the best practice of always using `Preconditions.checkArgument(...)` to verify the number of inputs is expected, for every public APIs such as fit/transform/setModelData.
Could you help add this check for both NaiveBayes and KMeans?
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.api.Expressions.$;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private Row[] trainData;
+ private Row[] predictData;
+ private Row[] expectedOutput;
+ private String featuresCol;
+ private String labelCol;
+ private String predictCol;
+ private String modelType;
+ private double smoothing;
+ private boolean isSaveLoad;
+ private String errorMessage;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ schema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.DOUBLE())
+ .column("f1", DataTypes.of(DenseVector.class))
+ .column("f2", DataTypes.DOUBLE())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+
+ predictData = trainData;
+
+ expectedOutput =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0, 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0, 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0, 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0, 10.0)
+ };
+
+ featuresCol = "features";
Review comment:
nits: It looks like this variable is only used once in this class. Does this need to be a class-level variable? It is preferred to use local variable instead of class-level variable if possible.
Same for other variables.
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.api.Expressions.$;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private Row[] trainData;
+ private Row[] predictData;
+ private Row[] expectedOutput;
+ private String featuresCol;
+ private String labelCol;
+ private String predictCol;
+ private String modelType;
+ private double smoothing;
+ private boolean isSaveLoad;
+ private String errorMessage;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ schema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.DOUBLE())
+ .column("f1", DataTypes.of(DenseVector.class))
+ .column("f2", DataTypes.DOUBLE())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+
+ predictData = trainData;
+
+ expectedOutput =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0, 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0, 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0, 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0, 10.0)
+ };
+
+ featuresCol = "features";
+ labelCol = "label";
+ predictCol = "predict";
+ modelType = "multinomial";
+ smoothing = 1.0;
+ isSaveLoad = false;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testNaiveBayes() throws Exception {
+ errorMessage = "normal test for Naive Bayes";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testPredictUnseenFeature() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if unseen feature values are met in prediction "
+ + "and the model type is categorical.";
+ predictData =
+ new Row[] {
+ Row.of(1., Vectors.dense(5, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(5, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testVectorWithDiffLen() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if length of feature vectors are not equal.";
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testVectorWithDiffLen2() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if length of feature vectors are not equal.";
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test
+ public void testSaveLoad() throws Exception {
+ errorMessage = "Naive Bayes should be able to save Model to filesystem and load correctly.";
+ isSaveLoad = true;
+ runAndCheck();
+ }
+
+ @Test
+ public void testGetModelData() throws Exception {
+ errorMessage =
+ "Naive Bayes should be able to provide model data with getModelData() method.";
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1.), 11.0), Row.of(2., Vectors.dense(2, 1.), 11.0)
+ };
+ NaiveBayesModel model = getModel();
+ NaiveBayesModelData expected =
+ new NaiveBayesModelData(
+ new HashMap[][] {
+ new HashMap[] {
+ new HashMap<Double, Double>() {
+ {
+ put(2.0, -0.6931471805599453);
+ put(1.0, -0.6931471805599453);
+ }
+ },
+ new HashMap<Double, Double>() {
+ {
+ put(1.0, 0.0);
+ }
+ },
+ },
+ },
+ new double[] {0.0},
+ new double[] {11.0});
+ NaiveBayesModelData actual =
+ NaiveBayesModelData.toDataStream(tEnv, model.getModelData()[0])
+ .executeAndCollect()
+ .next();
+ assertArrayEquals(errorMessage, expected.labels, actual.labels, 1e-5);
+ assertArrayEquals(errorMessage, expected.piArray, actual.piArray, 1e-5);
+ assertArrayEquals(errorMessage, expected.theta[0], actual.theta[0]);
+ }
+
+ @Test
+ public void testSetModelData() throws Exception {
+ errorMessage =
+ "Naive Bayes should be able to provide model data with getModelData() method.";
+
+ NaiveBayesModel modelA = getModel();
+
+ Table modelData = modelA.getModelData()[0];
+ NaiveBayesModel modelB = new NaiveBayesModel().setModelData(modelData);
+ ReadWriteUtils.updateExistingParams(modelB, modelA.getParamMap());
+
+ checkResult(modelB);
+ }
+
+ private void runAndCheck() throws Exception {
Review comment:
It is probably preferred to setup a best practice of writing unit tests for algorithms.
Any chance we can re-use the test pattern (e.g. `executeAndCollect`) used in `KMeansTest`?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+ private static final String broadcastModelKey = "NaiveBayesModelStream";
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+
+ List<String> colNames = new ArrayList<>(inputs[0].getResolvedSchema().getColumnNames());
+ colNames.add(predictionCol);
+
+ List<TypeInformation<?>> colTypes =
+ new ArrayList<>(inputs[0].getResolvedSchema().getColumnDataTypes())
+ .stream()
+ .map(
+ (Function<DataType, TypeInformation<?>>)
+ dataType ->
+ TypeInformation.of(
+ dataType.getConversionClass()))
+ .collect(Collectors.toList());
+ colTypes.add(TypeInformation.of(Object.class));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelStream =
+ NaiveBayesModelData.toDataStream(tEnv, modelTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ new RowTypeInfo(
+ colTypes.toArray(new TypeInformation[0]),
+ colNames.toArray(new String[0])),
+ new NaiveBayesPredictOp(new NaiveBayesPredictFunc(featuresCol)));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.toDataStream(tEnv, modelTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(String path) throws IOException {
Review comment:
Could you follow the example of KMeans and the Java doc in `Stage.java` to pass env as argument?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+ private static final String broadcastModelKey = "NaiveBayesModelStream";
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+
+ List<String> colNames = new ArrayList<>(inputs[0].getResolvedSchema().getColumnNames());
+ colNames.add(predictionCol);
+
+ List<TypeInformation<?>> colTypes =
Review comment:
Would it be simpler and more consistent to use the approach in `KMeansModel`?
KMeansModel uses something like this:
```
RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
RowTypeInfo outputTypeInfo =
new RowTypeInfo(
ArrayUtils.addAll(inputTypeInfo.getFieldTypes(), Types.INT),
ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
```
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.api.Expressions.$;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private Row[] trainData;
+ private Row[] predictData;
+ private Row[] expectedOutput;
+ private String featuresCol;
+ private String labelCol;
+ private String predictCol;
+ private String modelType;
+ private double smoothing;
+ private boolean isSaveLoad;
+ private String errorMessage;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ schema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.DOUBLE())
+ .column("f1", DataTypes.of(DenseVector.class))
+ .column("f2", DataTypes.DOUBLE())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+
+ predictData = trainData;
+
+ expectedOutput =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0, 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0, 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0, 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0, 10.0)
+ };
+
+ featuresCol = "features";
+ labelCol = "label";
+ predictCol = "predict";
+ modelType = "multinomial";
+ smoothing = 1.0;
+ isSaveLoad = false;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testNaiveBayes() throws Exception {
+ errorMessage = "normal test for Naive Bayes";
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testPredictUnseenFeature() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if unseen feature values are met in prediction "
+ + "and the model type is categorical.";
+ predictData =
+ new Row[] {
+ Row.of(1., Vectors.dense(5, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(5, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testVectorWithDiffLen() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if length of feature vectors are not equal.";
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test(expected = Exception.class)
+ public void testVectorWithDiffLen2() throws Exception {
+ errorMessage =
+ "Naive Bayes should throw exception if length of feature vectors are not equal.";
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1., 1., 1.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0)
+ };
+ runAndCheck();
+ }
+
+ @Test
+ public void testSaveLoad() throws Exception {
+ errorMessage = "Naive Bayes should be able to save Model to filesystem and load correctly.";
+ isSaveLoad = true;
+ runAndCheck();
+ }
+
+ @Test
+ public void testGetModelData() throws Exception {
+ errorMessage =
+ "Naive Bayes should be able to provide model data with getModelData() method.";
+ trainData =
+ new Row[] {
+ Row.of(1., Vectors.dense(1, 1.), 11.0), Row.of(2., Vectors.dense(2, 1.), 11.0)
+ };
+ NaiveBayesModel model = getModel();
+ NaiveBayesModelData expected =
+ new NaiveBayesModelData(
+ new HashMap[][] {
+ new HashMap[] {
+ new HashMap<Double, Double>() {
+ {
+ put(2.0, -0.6931471805599453);
+ put(1.0, -0.6931471805599453);
+ }
+ },
+ new HashMap<Double, Double>() {
+ {
+ put(1.0, 0.0);
+ }
+ },
+ },
+ },
+ new double[] {0.0},
+ new double[] {11.0});
+ NaiveBayesModelData actual =
+ NaiveBayesModelData.toDataStream(tEnv, model.getModelData()[0])
+ .executeAndCollect()
+ .next();
+ assertArrayEquals(errorMessage, expected.labels, actual.labels, 1e-5);
+ assertArrayEquals(errorMessage, expected.piArray, actual.piArray, 1e-5);
+ assertArrayEquals(errorMessage, expected.theta[0], actual.theta[0]);
+ }
+
+ @Test
+ public void testSetModelData() throws Exception {
+ errorMessage =
+ "Naive Bayes should be able to provide model data with getModelData() method.";
+
+ NaiveBayesModel modelA = getModel();
+
+ Table modelData = modelA.getModelData()[0];
+ NaiveBayesModel modelB = new NaiveBayesModel().setModelData(modelData);
+ ReadWriteUtils.updateExistingParams(modelB, modelA.getParamMap());
+
+ checkResult(modelB);
+ }
+
+ private void runAndCheck() throws Exception {
+ NaiveBayesModel model = getModel();
+ checkResult(model);
+ }
+
+ private NaiveBayesModel getModel() throws Exception {
+ Table trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(trainData)
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ schema)
+ .as("weight", "features", "label");
+
+ NaiveBayes estimator =
+ new NaiveBayes()
+ .setSmoothing(smoothing)
+ .setFeaturesCol(featuresCol)
+ .setLabelCol(labelCol)
+ .setPredictionCol(predictCol)
+ .setModelType(modelType);
+
+ if (isSaveLoad) {
+ String tempDir = Files.createTempDirectory("").toString();
+ estimator.save(tempDir);
+ env.execute();
+
+ estimator = NaiveBayes.load(tempDir);
+ }
+
+ NaiveBayesModel model = estimator.fit(trainTable);
+
+ if (isSaveLoad) {
+ String tempDir = Files.createTempDirectory("").toString();
+ model.save(tempDir);
+ env.execute();
+
+ model = NaiveBayesModel.load(tempDir);
+ }
+
+ return model;
+ }
+
+ private void checkResult(NaiveBayesModel model) {
+ Table predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(predictData)
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ schema)
+ .as("weight", "features", "label");
+
+ Table output =
+ model.transform(predictTable)[0].select(
+ $("weight"), $("features"), $("label"), $("predict"));
+
+ Object[] actualObjects = IteratorUtils.toArray(output.execute().collect());
+ Row[] actual = new Row[actualObjects.length];
+ for (int i = 0; i < actualObjects.length; i++) {
+ actual[i] = (Row) actualObjects[i];
+ }
+
+ assertEquals(errorMessage, getFrequencyMap(expectedOutput), getFrequencyMap(actual));
+ }
+
+ private static Map<Object, Integer> getFrequencyMap(Row[] rows) {
+ Map<Object, Integer> map = new HashMap<>();
+ for (Row row : rows) {
+ List<Object> list = toList(row);
+ map.put(list, map.getOrDefault(list, 0) + 1);
+ }
+ return map;
+ }
+
+ private static List<Object> toList(Row row) {
Review comment:
Would it be better to add a `List<Object> getFieldValues(Row row)` in `TableUtils`?
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+ private static final String broadcastModelKey = "NaiveBayesModelStream";
Review comment:
`broadcastModelKey` is an implementation detail and it is only used in `NaiveBayesPredictOp`. Can this be passed to `NaiveBayesPredictOp` directly?
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.api.Expressions.$;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private Row[] trainData;
+ private Row[] predictData;
+ private Row[] expectedOutput;
+ private String featuresCol;
+ private String labelCol;
+ private String predictCol;
+ private String modelType;
+ private double smoothing;
+ private boolean isSaveLoad;
+ private String errorMessage;
Review comment:
It is very rare to pass `errorMessage` via through class-level variable. Can you try to pass it to runAndCheck directly?
For example, class-level variables in KMeansTest are only written in `before()`.
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private static final long serialVersionUID = -4673084154965905629L;
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelTable;
+ private static final String broadcastModelKey = "NaiveBayesModelStream";
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+
+ List<String> colNames = new ArrayList<>(inputs[0].getResolvedSchema().getColumnNames());
+ colNames.add(predictionCol);
+
+ List<TypeInformation<?>> colTypes =
+ new ArrayList<>(inputs[0].getResolvedSchema().getColumnDataTypes())
+ .stream()
+ .map(
+ (Function<DataType, TypeInformation<?>>)
+ dataType ->
+ TypeInformation.of(
+ dataType.getConversionClass()))
+ .collect(Collectors.toList());
+ colTypes.add(TypeInformation.of(Object.class));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelStream =
+ NaiveBayesModelData.toDataStream(tEnv, modelTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ new RowTypeInfo(
+ colTypes.toArray(new TypeInformation[0]),
+ colNames.toArray(new String[0])),
+ new NaiveBayesPredictOp(new NaiveBayesPredictFunc(featuresCol)));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelTable).getTableEnvironment();
+
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.toDataStream(tEnv, modelTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(String path) throws IOException {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+ Source<NaiveBayesModelData, ?, ?> source =
+ FileSource.forRecordStreamFormat(
+ new NaiveBayesModelData.ModelDataStreamFormat(),
+ ReadWriteUtils.getDataPaths(path))
+ .build();
+ NaiveBayesModel model = ReadWriteUtils.loadStageParam(path);
+ DataStream<NaiveBayesModelData> modelData =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "modelData");
+ model.setModelData(NaiveBayesModelData.fromDataStream(tEnv, modelData));
+
+ return model;
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ @Override
+ public NaiveBayesModel setModelData(Table... inputs) {
+ modelTable = inputs[0];
+ return this;
+ }
+
+ @Override
+ public Table[] getModelData() {
+ return new Table[] {modelTable};
+ }
+
+ private static class NaiveBayesPredictOp
Review comment:
It is probably not necessary to have the algorithm name in this private static class. And it is preferred to make this name more meaningful.
How about changing this to something like `PredictLabelOperator`?
Same for other private classes in `NaiveBayes` and `NaiveBayesModel`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r750875949
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm using
+ * Bayes theorem based on independent assumption. It is an independent feature model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator<NaiveBayes, NaiveBayesModel>,
+ NaiveBayesParams<NaiveBayes> {
Review comment:
Got it. I have separated the two param classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r764636724
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/** A Model which classifies data using the model data computed by {@link NaiveBayes}. */
+public class NaiveBayesModel
+ implements Model<NaiveBayesModel>, NaiveBayesModelParams<NaiveBayesModel> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+ private Table modelDataTable;
+
+ public NaiveBayesModel() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public Table[] transform(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String predictionCol = getPredictionCol();
+ final String featuresCol = getFeaturesCol();
+ final String broadcastModelKey = "NaiveBayesModelStream";
+
+ RowTypeInfo inputTypeInfo = TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+ RowTypeInfo outputTypeInfo =
+ new RowTypeInfo(
+ ArrayUtils.addAll(
+ inputTypeInfo.getFieldTypes(), TypeInformation.of(Integer.class)),
+ ArrayUtils.addAll(inputTypeInfo.getFieldNames(), predictionCol));
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) modelDataTable).getTableEnvironment();
+ DataStream<NaiveBayesModelData> modelDataStream =
+ NaiveBayesModelData.getModelDataStream(modelDataTable);
+ DataStream<Row> input = tEnv.toDataStream(inputs[0]);
+
+ Map<String, DataStream<?>> broadcastMap = new HashMap<>();
+ broadcastMap.put(broadcastModelKey, modelDataStream);
+
+ Function<List<DataStream<?>>, DataStream<Row>> function =
+ dataStreams -> {
+ DataStream stream = dataStreams.get(0);
+ return stream.transform(
+ this.getClass().getSimpleName(),
+ outputTypeInfo,
+ new PredictLabelOperator(featuresCol, broadcastModelKey));
+ };
+ DataStream<Row> output =
+ BroadcastUtils.withBroadcastStream(
+ Collections.singletonList(input), broadcastMap, function);
+
+ Table outputTable = tEnv.fromDataStream(output);
+
+ return new Table[] {outputTable};
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ String dataPath = ReadWriteUtils.getDataPath(path);
+ FileSink<NaiveBayesModelData> sink =
+ FileSink.forRowFormat(
+ new Path(dataPath), new NaiveBayesModelData.ModelDataEncoder())
+ .withRollingPolicy(OnCheckpointRollingPolicy.build())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+ NaiveBayesModelData.getModelDataStream(modelDataTable).sinkTo(sink);
+
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayesModel load(StreamExecutionEnvironment env, String path)
+ throws IOException {
+ Source<NaiveBayesModelData, ?, ?> source =
+ FileSource.forRecordStreamFormat(
+ new NaiveBayesModelData.ModelDataStreamFormat(),
+ ReadWriteUtils.getDataPaths(path))
+ .build();
+ NaiveBayesModel model = ReadWriteUtils.loadStageParam(path);
+ DataStream<NaiveBayesModelData> modelData =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "modelData");
+ model.setModelData(NaiveBayesModelData.getModelDataTable(modelData));
+
+ return model;
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ @Override
+ public NaiveBayesModel setModelData(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+ modelDataTable = inputs[0];
+ return this;
+ }
+
+ @Override
+ public Table[] getModelData() {
+ return new Table[] {modelDataTable};
+ }
+
+ private static class PredictLabelOperator
+ extends AbstractUdfStreamOperator<Row, AbstractRichFunction>
+ implements OneInputStreamOperator<Row, Row> {
+ private final String featuresCol;
+ private final String broadcastModelKey;
+
+ public PredictLabelOperator(String featuresCol, String broadcastModelKey) {
+ super(new AbstractRichFunction() {});
+ this.featuresCol = featuresCol;
+ this.broadcastModelKey = broadcastModelKey;
+ }
+
+ @Override
+ public void processElement(StreamRecord<Row> streamRecord) {
+ Row inputRow = streamRecord.getValue();
+ NaiveBayesModelData modelData =
Review comment:
OK. I'll do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r764655403
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModelData.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The model data of {@link NaiveBayesModel}.
+ *
+ * <p>This class also provides methods to convert model data between Table and Datastream, and
+ * classes to save/load model data.
+ */
+public class NaiveBayesModelData implements Serializable {
+ public final Map<Double, Double>[][] theta;
Review comment:
OK. I'll add these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r757211597
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Table trainTable;
+ private Table predictTable;
+ private Map<Vector, Integer> expectedOutput;
+ private NaiveBayes estimator;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ Schema trainSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .column("f1", DataTypes.INT())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 0.), 11),
+ Row.of(Vectors.dense(1, 0), 10),
+ Row.of(Vectors.dense(1, 1.), 10))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ trainSchema)
+ .as("features", "label");
+
+ Schema predictSchema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.of(DenseVector.class))
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ predictTable =
+ tEnv.fromDataStream(
+ env.fromElements(
+ Row.of(Vectors.dense(0, 1.)),
+ Row.of(Vectors.dense(0, 0.)),
+ Row.of(Vectors.dense(1, 0)),
+ Row.of(Vectors.dense(1, 1.)))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.noWatermarks()),
+ predictSchema)
+ .as("features");
+
+ expectedOutput =
+ new HashMap<Vector, Integer>() {
+ {
+ put(Vectors.dense(0, 1.), 11);
+ put(Vectors.dense(0, 0.), 11);
+ put(Vectors.dense(1, 0.), 10);
+ put(Vectors.dense(1, 1.), 10);
+ }
+ };
+
+ estimator =
+ new NaiveBayes()
+ .setSmoothing(1.0)
+ .setFeaturesCol("features")
+ .setLabelCol("label")
+ .setPredictionCol("predict")
+ .setModelType("multinomial");
+ }
+
+ /**
+ * Executes a given table and collect its results. Results are returned as a map whose key is
+ * the feature, value is the prediction result.
+ *
+ * @param table A table to be executed and to have its result collected
+ * @param featuresCol Name of the column in the table that contains the features
+ * @param predictionCol Name of the column in the table that contains the prediction result
+ * @return A map containing the collected results
+ */
+ private static Map<Vector, Integer> executeAndCollect(
+ Table table, String featuresCol, String predictionCol) {
+ Map<Vector, Integer> map = new HashMap<>();
+ for (CloseableIterator<Row> it = table.execute().collect(); it.hasNext(); ) {
+ Row row = it.next();
+ map.put(
+ (Vector) row.getField(featuresCol),
+ ((Number) row.getField(predictionCol)).intValue());
+ }
+
+ return map;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testFitAndPredict() throws Exception {
Review comment:
I think it is better to separate `testFitAndPredict` and `testFeaturePredictionParam`, like in `KMeansTest`. I'll add a new `testFeaturePredictionParam` in this class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755724605
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.ml.util.StageTestUtils;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+
+import static org.apache.flink.table.api.Expressions.$;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** Tests {@link NaiveBayes} and {@link NaiveBayesModel}. */
+public class NaiveBayesTest {
+ private StreamExecutionEnvironment env;
+ private StreamTableEnvironment tEnv;
+ private Schema schema;
+ private List<Row> trainData;
+ private List<Row> predictData;
+ private List<Row> expectedOutput;
+ private boolean isSaveLoad;
+
+ @Before
+ public void before() {
+ Configuration config = new Configuration();
+ config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
+ env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ env.setParallelism(4);
+ env.enableCheckpointing(100);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ tEnv = StreamTableEnvironment.create(env);
+
+ schema =
+ Schema.newBuilder()
+ .column("f0", DataTypes.DOUBLE())
+ .column("f1", DataTypes.of(DenseVector.class))
+ .column("f2", DataTypes.DOUBLE())
+ .columnByMetadata("rowtime", "TIMESTAMP_LTZ(3)")
+ .watermark("rowtime", "SOURCE_WATERMARK()")
+ .build();
+
+ trainData =
+ Arrays.asList(
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0));
+
+ predictData = trainData;
+
+ expectedOutput =
+ Arrays.asList(
+ Row.of(1., Vectors.dense(1, 1., 1., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(1, 1., 0., 1., 2.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1., 3.), 11.0, 11.0),
+ Row.of(1., Vectors.dense(2, 0., 1., 1.5, 2.), 11.0, 11.0),
+ Row.of(2., Vectors.dense(3, 1.5, 1., 0.5, 3.), 10.0, 10.0),
+ Row.of(1., Vectors.dense(1, 1., 1.5, 0., 1.), 10.0, 10.0),
+ Row.of(2., Vectors.dense(4, 1., 1., 0., 1.), 10.0, 10.0));
+
+ isSaveLoad = false;
+ }
+
+ @Test
+ public void testParam() {
+ NaiveBayes estimator = new NaiveBayes();
+
+ assertEquals("features", estimator.getFeaturesCol());
+ assertEquals("label", estimator.getLabelCol());
+ assertEquals("multinomial", estimator.getModelType());
+ assertEquals("prediction", estimator.getPredictionCol());
+ assertEquals(1.0, estimator.getSmoothing(), 1e-5);
+
+ estimator
+ .setFeaturesCol("test_feature")
+ .setLabelCol("test_label")
+ .setPredictionCol("test_prediction")
+ .setSmoothing(2.0);
+
+ assertEquals("test_feature", estimator.getFeaturesCol());
+ assertEquals("test_label", estimator.getLabelCol());
+ assertEquals("test_prediction", estimator.getPredictionCol());
+ assertEquals(2.0, estimator.getSmoothing(), 1e-5);
+
+ NaiveBayesModel model = new NaiveBayesModel();
+
+ assertEquals("features", model.getFeaturesCol());
+ assertEquals("multinomial", model.getModelType());
+ assertEquals("prediction", model.getPredictionCol());
+
+ model.setFeaturesCol("test_feature").setPredictionCol("test_prediction");
+
+ assertEquals("test_feature", model.getFeaturesCol());
+ assertEquals("test_prediction", model.getPredictionCol());
+ }
+
+ @Test
+ public void testNaiveBayes() throws Exception {
Review comment:
OK. I'll fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
lindong28 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755625715
##########
File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * <p>See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+ implements Estimator<NaiveBayes, NaiveBayesModel>, NaiveBayesParams<NaiveBayes> {
+ private final Map<Param<?>, Object> paramMap = new HashMap<>();
+
+ public NaiveBayes() {
+ ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+ }
+
+ @Override
+ public NaiveBayesModel fit(Table... inputs) {
+ Preconditions.checkArgument(inputs.length == 1);
+
+ final String featuresCol = getFeaturesCol();
+ final String labelCol = getLabelCol();
+ final double smoothing = getSmoothing();
+
+ StreamTableEnvironment tEnv =
+ (StreamTableEnvironment) ((TableImpl) inputs[0]).getTableEnvironment();
+ DataStream<Tuple2<Vector, Double>> input =
+ tEnv.toDataStream(inputs[0])
+ .map(
+ new MapFunction<Row, Tuple2<Vector, Double>>() {
+ @Override
+ public Tuple2<Vector, Double> map(Row row) throws Exception {
+ return new Tuple2<>(
+ (Vector) row.getField(featuresCol),
+ (Double) row.getField(labelCol));
+ }
+ });
+
+ DataStream<NaiveBayesModelData> naiveBayesModel =
+ input.flatMap(new FlattenFunction())
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple3<>(value.f0, value.f1, value.f2))
+ .window(EndOfStreamWindows.get())
+ .reduce(
+ (ReduceFunction<Tuple4<Double, Integer, Double, Double>>)
+ (t0, t1) -> {
+ t0.f3 += t1.f3;
+ return t0;
+ })
+ .keyBy(
+ (KeySelector<Tuple4<Double, Integer, Double, Double>, Object>)
+ value -> new Tuple2<>(value.f0, value.f1))
+ .window(EndOfStreamWindows.get())
+ .aggregate(new ValueMapFunction())
+ .keyBy(
+ (KeySelector<
+ Tuple4<
+ Double,
+ Integer,
+ Map<Double, Double>,
+ Double>,
+ Object>)
+ value -> value.f0)
+ .window(EndOfStreamWindows.get())
+ .aggregate(new MapArrayFunction())
+ .windowAll(EndOfStreamWindows.get())
+ .apply(new GenerateModelFunction(smoothing));
+
+ NaiveBayesModel model =
+ new NaiveBayesModel()
+ .setModelData(NaiveBayesModelData.fromDataStream(tEnv, naiveBayesModel));
+ ReadWriteUtils.updateExistingParams(model, paramMap);
+ return model;
+ }
+
+ @Override
+ public void save(String path) throws IOException {
+ ReadWriteUtils.saveMetadata(this, path);
+ }
+
+ public static NaiveBayes load(StreamExecutionEnvironment env, String path) throws IOException {
+ return ReadWriteUtils.loadStageParam(path);
+ }
+
+ @Override
+ public Map<Param<?>, Object> getParamMap() {
+ return paramMap;
+ }
+
+ /**
+ * Function to convert each column into tuples of label, feature column index, feature value,
+ * weight.
+ */
+ private static class FlattenFunction
+ implements FlatMapFunction<
+ Tuple2<Vector, Double>, Tuple4<Double, Integer, Double, Double>> {
Review comment:
Would it be better to let `label` be `Integer` instead of `Double`, given that the label in `NaiveBayes` should always be `Integer`?
Note that the label value is used as the grouping key in the program. I am not sure if there will misclassification due to precision error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r755747666
##########
File path: flink-ml-lib/src/test/java/org/apache/flink/ml/util/StageTestUtils.java
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.util;
+
+import org.apache.flink.ml.api.core.Stage;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import java.lang.reflect.Method;
+import java.nio.file.Files;
+
+/** Utility methods for testing stages. */
+public class StageTestUtils {
+ /**
+ * Saves a stage to filesystem and reloads it with the static load() method a stage must
+ * implement.
+ */
+ public static <T extends Stage<T>> T saveAndReload(StreamExecutionEnvironment env, T stage)
+ throws Exception {
+ String tempDir = Files.createTempDirectory("").toString();
Review comment:
OK. I'll do it.
The common practice in Flink tests is to initialize the `TemporaryFolder` in each test class, and calls for initialization in `@Before` methods. Since the usage of `TemporaryFolder` in our case is merely in static utility functions, I think we can just put it as a static member of that class and initializes in static context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on pull request #32: [FLINK-24817] Add Naive Bayes implementation
Posted by GitBox <gi...@apache.org>.
lindong28 commented on pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#issuecomment-980639795
Thanks for the update! LGTM. @gaoyunhaii could you help review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org