You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by GitBox <gi...@apache.org> on 2019/12/23 14:02:55 UTC
[GitHub] [submarine] pingsutw opened a new pull request #132: SUBMARINE-296
[SDK] Submarine pipeline example
pingsutw opened a new pull request #132: SUBMARINE-296 [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132
### What is this PR for?
quick start deep-learning based CTR example of submarine pipeline
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-296
### How should this be tested?
https://travis-ci.org/pingsutw/hadoop-submarine/builds/628696018
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #132:
SUBMARINE-296 [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #132: SUBMARINE-296 [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r361043839
##########
File path: submarine-sdk/pysubmarine/example/deepFM.json
##########
@@ -0,0 +1,31 @@
+{
+ "input": {
+ "train_data": "./data/tr.libsvm",
Review comment:
Thanks for your contribution! @pingsutw It looks good to me. One minor suggestion, can we use the hyphen instead of underscore? The underscore character seems more preferred from a developer view and is not user-friendly to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] wangdatan commented on issue #132: SUBMARINE-296. [SDK]
Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
wangdatan commented on issue #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#issuecomment-587096514
@pingsutw, thanks for working on this!
I took a look at the patch. Some comments/suggestions:
First of all, I think this is a decent example to get data scientists started and allow us to add more models in the future. I like the simplicity brought by this example.
**1) Regarding source code organization:**
It is not very clear to me about what is entry point during review. I suggest following changes to the source code structure:
- Put all deepFM.json/run_deepFM.py and data into submarine-sdk/pysubmarine/examples/deepfm
- Move all deep_fm-related code into pysubmarine/submarine/ml to pysubmarine/submarine/model/deep_fm
- There're already some common folders inside pysubmarine/submarine, like `utils`, etc. I think we should use them instead of creating new ones.
- abstractModel can be moved to pysubmarine/submarine/model
**2) Regarding code style**
There're mixed style of namings in classes, functions and variables (like abstractModel.py, test_utils.py), we should make them consistent.
**3) for model/deepFM.py**
I'm wondering if it is from the original deep FM project, if yes I think we should add a comment to the file and point to the original Github link.
4) Add a README.md file into submarine-sdk/pysubmarine/example
It is fine to make it super simple now, but it will be helpful to improve our examples library in the future.
Thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r383052667
##########
File path: submarine-sdk/pysubmarine/submarine/ml/parameters.py
##########
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+LIBSVM = "libsvm"
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r383634106
##########
File path: submarine-sdk/pysubmarine/example/deepfm/README.md
##########
@@ -0,0 +1,57 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+### Running Examples
+To run the examples here, you need to:
+- Build a Python virtual environment with pysubmarine installed
+- Install Submarine 0.3.0+
+### Running DeepFM on a local machine
+1. Create a JSON configuration file containing train,valid and test data, model parameters,
+metrics, save model path, resources. e.g. [deepfm.json](./deepfm.json)
+
+2. Install submarine python bindings by setup.py:
+```
+python ./submarine/submarine-sdk/setup.py install
+```
+3. Run DeepFM
+```
+python run_deepfm.py -conf=deepfm.json
+```
+### Running DeepFM on Submarine
+1. Upload data to a shared file system like hdfs, s3.
+
+2. Create a JSON configuration file for distributed training. e.g. [deepfm_distributed.json](./deepfm_distributed.json)
+
+3. Submit Job
+```
+SUBMARINE_VERSION=0.4.0
+SUBMARINE_HADOOP_VERSION=2.9
+
+java -cp $(${HADOOP_COMMON_HOME}/bin/hadoop classpath --glob):submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar:${HADOOP_CONF_PATH} \
+ org.apache.submarine.client.cli.Cli job run --name deepfm-job-001 \
+ --framework tensorflow \
+ --verbose \
+ --input_path "" \
+ --num_workers 2 \
+ --worker_resources memory=4G,vcores=4 \
+ --num_ps 1 \
+ --ps_resources memory=4G,vcores=4 \
+ --worker_launch_cmd "myvenv.zip/venv/bin/python run_deepfm.py -conf=deepfm_distributed.json" \
+ --ps_launch_cmd "myvenv.zip/venv/bin/python run_deepfm.py -conf=deepfm_distributed.json" \
+ --insecure \
+ --conf tony.containers.resources=myvenv.zip#archive,submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar,deepfm_distributed.json,run_deepfm.py \
+ --conf tony.chief.instances=1 \
Review comment:
Is the "--config tony.*=value" necessary?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r380485882
##########
File path: submarine-sdk/pysubmarine/submarine/ml/parameters.py
##########
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+LIBSVM = "libsvm"
+
+defaultParameters = {
+ "output": {
+ "save_model_dir": "./experiment",
+ "metric": "auc"
+ },
+ "training": {
+ "batch_size": 512,
+ "field_size": 39,
+ "num_epochs": 3,
+ "feature_size": 117581,
+ "embedding_size": 256,
+ "learning_rate": 0.0005,
+ "batch_norm_decay": 0.9,
+ "l2_reg": 0.0001,
+ "deep_layers": "400,400,400",
+ "dropout": "0.3,0.3,0.3",
+ "batch_norm": "false",
+ "optimizer": "adam",
+ "log_steps": 10,
+ "num_threads": 4,
+ "num_gpu": 0,
+ "seed": 77,
+ "mode": "local"
+ }
+}
Review comment:
They cover all other models, except `deep_layers`
- In my opinion, the advantage of leaving all parameters in JSON is that users can easily tune their parameters in one file, and the JSON spec will be refined in the future.
- Actually, if we specify parameters that not include in Deepfm, it will not affect training.
The model will only get the parameters they required.
- I will add a doc for this JSON spec so that users can know what kind of parameters that need to write at the beginning
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yanboliang commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
yanboliang commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r380486678
##########
File path: submarine-sdk/pysubmarine/example/deepFM.json
##########
@@ -0,0 +1,31 @@
+{
+ "input": {
+ "train_data": "./data/tr.libsvm",
+ "valid_data": "./data/va.libsvm",
+ "test_data": "./data/te.libsvm",
+ "type": "libsvm"
+ },
+ "output": {
+ "save_model_dir": "./experiment",
+ "metric": "auc"
+ },
+ "training": {
+ "batch_size" : 512,
+ "field_size": 39,
+ "num_epochs": 3,
+ "feature_size": 117581,
+ "embedding_size": 256,
+ "learning_rate": 0.0005,
+ "batch_norm_decay": 0.9,
+ "l2_reg": 0.0001,
+ "deep_layers": "400,400,400",
+ "dropout": "0.3,0.3,0.3",
+ "batch_norm": false,
+ "optimizer": "adam",
+ "log_steps": 10,
+ "num_threads": 4,
+ "num_gpu": 0,
+ "seed": 77,
+ "mode": "local"
Review comment:
Can we reorg the ```training``` parameters into two parts: ```algorithm``` and ```system```. The ```algorithm``` part should be more friendly to data scientists and machine learning engineers, and the ```system``` part would be more advanced.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yanboliang commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
yanboliang commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r380506768
##########
File path: submarine-sdk/pysubmarine/example/deepFM.json
##########
@@ -0,0 +1,31 @@
+{
+ "input": {
+ "train_data": "./data/tr.libsvm",
Review comment:
I'd suggest to use array to store training/eval/test dataset. For example, machine learning engineers would try to train model on the dataset from 01/01/2020 to 01/14/2020, and evaluate model's performance on 01/15/2020. This is the most common cases and it would be great we support it at the beginning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yanboliang commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
yanboliang commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r380495415
##########
File path: submarine-sdk/pysubmarine/example/deepFM.json
##########
@@ -0,0 +1,31 @@
+{
Review comment:
I would suggest to organize the structure in the following way:
* submarine-sdk/pysubmarine/example/deepFM/run_deepFM.py // Entry point
* submarine-sdk/pysubmarine/example/deepFM/deepFM.json // Configuration
* submarine-sdk/pysubmarine/example/data/deepFM/***.libsvm // Data
We can follow this structure to add more models/workflows. ```submarine-sdk/pysubmarine/example/data``` can be mapped to HDFS and then users can directly access the data via HDFS API.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on issue #132: SUBMARINE-296. [SDK]
Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on issue #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#issuecomment-592842482
Will merge if no more comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r381078160
##########
File path: submarine-sdk/pysubmarine/example/deepFM.json
##########
@@ -0,0 +1,31 @@
+{
Review comment:
yes, if we use hdfs:// prefix for a path, we can directly access data in hdfs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yanboliang commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
yanboliang commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r380497727
##########
File path: submarine-sdk/pysubmarine/submarine/ml/model/deepFM.py
##########
@@ -0,0 +1,236 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import tensorflow as tf
+import submarine.ml.utils
+from submarine.ml.parameters import defaultParameters
+from submarine.ml.utils import get_TFConfig, get_from_registry, sanity_checks
+from submarine.ml.model.abstractModel import abstractModel
+from submarine.ml.registries import input_fn_registry
+
+logger = logging.getLogger(__name__)
+
+
+def batch_norm_layer(x, train_phase, scope_bn, batch_norm_decay):
+ bn_train = tf.contrib.layers.batch_norm(x, decay=batch_norm_decay, center=True, scale=True,
+ updates_collections=None, is_training=True,
+ reuse=None, scope=scope_bn)
+ bn_infer = tf.contrib.layers.batch_norm(x, decay=batch_norm_decay, center=True, scale=True,
+ updates_collections=None, is_training=False,
+ reuse=True, scope=scope_bn)
+ z = tf.cond(tf.cast(train_phase, tf.bool), lambda: bn_train, lambda: bn_infer)
+ return z
+
+
+def model_fn(features, labels, mode, params):
+ field_size = params["training"]["field_size"]
+ feature_size = params["training"]["feature_size"]
+ embedding_size = params["training"]["embedding_size"]
+ l2_reg = params["training"]["l2_reg"]
+ learning_rate = params["training"]["learning_rate"]
+ batch_norm = params["training"]["batch_norm"]
+ batch_norm_decay = params["training"]["batch_norm_decay"]
+ optimizer = params["training"]["optimizer"]
+ seed = params["training"]["seed"]
+ metric = params['output']['metric']
+
+ layers = list(map(int, params["training"]["deep_layers"].split(',')))
+ dropout = list(map(float, params["training"]["dropout"].split(',')))
+
+ tf.set_random_seed(seed)
+ fm_bias = tf.get_variable(name='fm_bias', shape=[1], initializer=tf.constant_initializer(0.0))
+ fm_weight = tf.get_variable(name='fm_weight', shape=[feature_size],
+ initializer=tf.glorot_normal_initializer())
+ fm_vector = tf.get_variable(name='fm_vector', shape=[feature_size, embedding_size],
+ initializer=tf.glorot_normal_initializer())
+
+ with tf.variable_scope("Feature"):
+ feat_ids = features['feat_ids']
+ feat_ids = tf.reshape(feat_ids, shape=[-1, field_size])
+ feat_vals = features['feat_vals']
+ feat_vals = tf.reshape(feat_vals, shape=[-1, field_size])
+
+ with tf.variable_scope("First_order"):
+ feat_weights = tf.nn.embedding_lookup(fm_weight, feat_ids)
+ y_w = tf.reduce_sum(tf.multiply(feat_weights, feat_vals), 1)
+
+ with tf.variable_scope("Second_order"):
+ embeddings = tf.nn.embedding_lookup(fm_vector, feat_ids)
+ feat_vals = tf.reshape(feat_vals, shape=[-1, field_size, 1])
+ embeddings = tf.multiply(embeddings, feat_vals)
+ sum_square = tf.square(tf.reduce_sum(embeddings, 1))
+ square_sum = tf.reduce_sum(tf.square(embeddings), 1)
+ y_v = 0.5 * tf.reduce_sum(tf.subtract(sum_square, square_sum), 1)
+
+ with tf.variable_scope("Deep-part"):
+ if batch_norm:
+ if mode == tf.estimator.ModeKeys.TRAIN:
+ train_phase = True
+ else:
+ train_phase = False
+
+ deep_inputs = tf.reshape(embeddings, shape=[-1, field_size * embedding_size])
+ for i in range(len(layers)):
+ deep_inputs = tf.contrib.layers.fully_connected(
+ inputs=deep_inputs, num_outputs=layers[i],
+ weights_regularizer=tf.contrib.layers.l2_regularizer(l2_reg),
+ scope='mlp%d' % i)
+ if batch_norm:
+ deep_inputs = batch_norm_layer(
+ deep_inputs, train_phase=train_phase,
+ scope_bn='bn_%d' % i, batch_norm_decay=batch_norm_decay)
+ if mode == tf.estimator.ModeKeys.TRAIN:
+ deep_inputs = tf.nn.dropout(deep_inputs, keep_prob=dropout[i])
+
+ y_deep = tf.contrib.layers.fully_connected(
+ inputs=deep_inputs, num_outputs=1, activation_fn=tf.identity,
+ weights_regularizer=tf.contrib.layers.l2_regularizer(l2_reg),
+ scope='deep_out')
+ y_d = tf.reshape(y_deep, shape=[-1])
+
+ with tf.variable_scope("DeepFM-out"):
+ y_bias = fm_bias * tf.ones_like(y_d, dtype=tf.float32)
+ y = y_bias + y_w + y_v + y_d
+ pred = tf.sigmoid(y)
+
+ predictions = {"probabilities": pred}
+ export_outputs = {
+ tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:
+ tf.estimator.export.PredictOutput(predictions)}
+ # Provide an estimator spec for `ModeKeys.PREDICT`
+ if mode == tf.estimator.ModeKeys.PREDICT:
+ return tf.estimator.EstimatorSpec(
+ mode=mode,
+ predictions=predictions,
+ export_outputs=export_outputs)
+
+ with tf.name_scope("Loss"):
+ loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=y, labels=labels)) + \
+ l2_reg * tf.nn.l2_loss(fm_weight) + l2_reg * tf.nn.l2_loss(fm_vector)
+
+ # Provide an estimator spec for `ModeKeys.EVAL`
+ eval_metric_ops = {}
+ if metric == 'auc':
+ eval_metric_ops['auc'] = tf.metrics.auc(labels, pred)
+ else:
+ raise TypeError("Can not find loss_type :", params['training']['loss_type'])
+
+ if mode == tf.estimator.ModeKeys.EVAL:
+ return tf.estimator.EstimatorSpec(
+ mode=mode,
+ predictions=predictions,
+ loss=loss,
+ eval_metric_ops=eval_metric_ops)
+
+ with tf.name_scope("Optimizer"):
+ if optimizer == 'adam':
+ op = tf.train.AdamOptimizer(learning_rate=learning_rate,
+ beta1=0.9, beta2=0.999, epsilon=1e-8)
+ elif optimizer == 'adagrad':
+ op = tf.train.AdagradOptimizer(
+ learning_rate=learning_rate, initial_accumulator_value=1e-8)
+ elif optimizer == 'momentum':
+ op = tf.train.MomentumOptimizer(learning_rate=learning_rate, momentum=0.95)
+ elif optimizer == 'ftrl':
+ op = tf.train.FtrlOptimizer(learning_rate)
+ else:
+ raise TypeError("Can not find optimizer :", optimizer)
+
+ train_op = op.minimize(loss, global_step=tf.train.get_global_step())
+
+ # Provide an estimator spec for `ModeKeys.TRAIN` modes
+ if mode == tf.estimator.ModeKeys.TRAIN:
+ return tf.estimator.EstimatorSpec(
+ mode=mode,
+ predictions=predictions,
+ loss=loss,
+ train_op=train_op)
+
+
+class deepFM(abstractModel):
+ def __init__(self, model_params=None, json_path=None):
+ """
+ Create a tensorflow DeepFM model
+ :param model_params: defines the different
+ parameters of the model, features, preprocessing and training
+ :param json_path: The JSON file that specifies the model parameters.
+ """
+ super().__init__()
+ self.model_params = submarine.ml.utils.merge_dicts(model_params, defaultParameters)
+ self.model_params = submarine.ml.utils.merge_json(json_path, self.model_params)
+ sanity_checks(self.model_params)
+ logging.info("Model parameters : %s", self.model_params)
+ self.input_type = self.model_params['input']['type']
+ self.model_dir = self.model_params['output']['save_model_dir']
+ self.config = get_TFConfig(self.model_params)
+ self.model = tf.estimator.Estimator(
+ model_fn=model_fn, model_dir=self.model_dir,
+ params=self.model_params, config=self.config)
+
+ def train(self, train_input_fn=None, eval_input_fn=None, **kwargs):
+ """
+ Trains a pre-defined tensorflow estimator model with given training data
+ :param train_input_fn: A function that provides input data for training.
+ :param eval_input_fn: A function that provides input data for evaluating.
+ :return: None
+ """
+ if train_input_fn is None:
+ train_input_fn = get_from_registry(
+ self.input_type, input_fn_registry)(
+ filepath=self.model_params['input']['train_data'],
+ **self.model_params['training'])
+ if eval_input_fn is None:
+ eval_input_fn = get_from_registry(
+ self.input_type, input_fn_registry)(
+ filepath=self.model_params['input']['valid_data'],
+ **self.model_params['training'])
+
+ train_spec = tf.estimator.TrainSpec(
+ input_fn=train_input_fn)
+ eval_spec = tf.estimator.EvalSpec(
+ input_fn=eval_input_fn)
+ tf.estimator.train_and_evaluate(self.model, train_spec, eval_spec)
+
+ def evaluate(self, eval_input_fn=None, **kwargs):
+ """
+ Evaluates a pre-defined Tensorflow estimator model with given evaluate data
+ :param eval_input_fn: A function that provides input data for evaluating.
+ :return: A dict containing the evaluation metrics specified in `eval_input_fn` keyed by
+ name, as well as an entry `global_step` which contains the value of the
+ global step for which this evaluation was performed
+ """
+ if eval_input_fn is None:
+ eval_input_fn = get_from_registry(
+ self.input_type, input_fn_registry)(
+ filepath=self.model_params['input']['valid_data'],
+ **self.model_params['training'])
+
+ return self.model.evaluate(input_fn=eval_input_fn, **kwargs)
+
+ def predict(self, test_input_fn=None, **kwargs):
Review comment:
Can you let me know where you refer the name "test_input_fn"? I'd prefer ```predict_input_fn``` or ```inference_input_fn```.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on issue #132: SUBMARINE-296. [SDK]
Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
pingsutw commented on issue #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#issuecomment-588035158
Very thanks @wangdatan @tangzhankun @yanboliang for the review and suggestion.
I'm going to update the patch these days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on issue #132: SUBMARINE-296. [SDK]
Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
pingsutw commented on issue #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#issuecomment-587273947
@wangdatan Thank you for the review.
1. About code organization and code style, thanks for the suggestions, I will update it.
2. There are some projects about CTR prediction, DeepFm is one of the machine learning algorithms in CTR. I refer to the part of them. I have made sure that the projects I refer are under the Apache or MIT license. I could add the links in the comment.
3. I will add a doc about how to submit a job from submarine CLI or YAML
4. Yes, the example is very simple now. There are many things to improve this library, support other algorithms, data format, doc, etc. I will keep working on it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] asfgit closed pull request #132: SUBMARINE-296. [SDK]
Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r381076545
##########
File path: submarine-sdk/pysubmarine/example/deepFM.json
##########
@@ -0,0 +1,31 @@
+{
+ "input": {
+ "train_data": "./data/tr.libsvm",
+ "valid_data": "./data/va.libsvm",
+ "test_data": "./data/te.libsvm",
+ "type": "libsvm"
+ },
+ "output": {
+ "save_model_dir": "./experiment",
+ "metric": "auc"
+ },
+ "training": {
+ "batch_size" : 512,
+ "field_size": 39,
+ "num_epochs": 3,
+ "feature_size": 117581,
+ "embedding_size": 256,
+ "learning_rate": 0.0005,
+ "batch_norm_decay": 0.9,
+ "l2_reg": 0.0001,
+ "deep_layers": "400,400,400",
+ "dropout": "0.3,0.3,0.3",
+ "batch_norm": false,
+ "optimizer": "adam",
+ "log_steps": 10,
+ "num_threads": 4,
+ "num_gpu": 0,
+ "seed": 77,
+ "mode": "local"
Review comment:
> Can we reorg the `training` parameters into two parts: `algorithm` and `system`. The `algorithm` part should be more friendly to data scientists and machine learning engineers, and the `system` part would be more advanced.
Agree, it will be more user-friendly for users, I will update it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yanboliang commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
yanboliang commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r380505075
##########
File path: submarine-sdk/pysubmarine/example/deepFM.json
##########
@@ -0,0 +1,31 @@
+{
+ "input": {
+ "train_data": "./data/tr.libsvm",
+ "valid_data": "./data/va.libsvm",
+ "test_data": "./data/te.libsvm",
+ "type": "libsvm"
+ },
+ "output": {
+ "save_model_dir": "./experiment",
+ "metric": "auc"
+ },
+ "training": {
+ "batch_size" : 512,
+ "field_size": 39,
+ "num_epochs": 3,
+ "feature_size": 117581,
+ "embedding_size": 256,
+ "learning_rate": 0.0005,
+ "batch_norm_decay": 0.9,
+ "l2_reg": 0.0001,
+ "deep_layers": "400,400,400",
+ "dropout": "0.3,0.3,0.3",
+ "batch_norm": false,
+ "optimizer": "adam",
+ "log_steps": 10,
+ "num_threads": 4,
+ "num_gpu": 0,
+ "seed": 77,
+ "mode": "local"
Review comment:
Can we add a ```distributed``` running script? I think many users who are trying submarine is interested to use distributed training. It would be great to have a distributed configuration file to let them get the model running quickly and understand which are the parameters need to be set if running distributed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] wangdatan commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
wangdatan commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r380319472
##########
File path: submarine-sdk/pysubmarine/submarine/ml/parameters.py
##########
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+LIBSVM = "libsvm"
+
+defaultParameters = {
+ "output": {
+ "save_model_dir": "./experiment",
+ "metric": "auc"
+ },
+ "training": {
+ "batch_size": 512,
+ "field_size": 39,
+ "num_epochs": 3,
+ "feature_size": 117581,
+ "embedding_size": 256,
+ "learning_rate": 0.0005,
+ "batch_norm_decay": 0.9,
+ "l2_reg": 0.0001,
+ "deep_layers": "400,400,400",
+ "dropout": "0.3,0.3,0.3",
+ "batch_norm": "false",
+ "optimizer": "adam",
+ "log_steps": 10,
+ "num_threads": 4,
+ "num_gpu": 0,
+ "seed": 77,
+ "mode": "local"
+ }
+}
Review comment:
How common this parameter to cover other models? If it is common enough, we can leave it either here or move it to model. If it is more specific to deepFM model, we shoud move it to model/deep_fm.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r380433869
##########
File path: submarine-sdk/pysubmarine/example/deepFM.json
##########
@@ -0,0 +1,31 @@
+{
+ "input": {
+ "train_data": "./data/tr.libsvm",
Review comment:
I see. Let's keep it underscore then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw opened a new pull request #132: SUBMARINE-296.
[SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
pingsutw opened a new pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132
### What is this PR for?
quick start deep-learning based CTR example of submarine pipeline
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-296
### How should this be tested?
https://travis-ci.org/pingsutw/hadoop-submarine/builds/628696018
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on issue #132: SUBMARINE-296. [SDK]
Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
pingsutw commented on issue #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#issuecomment-590132189
To sum up, update include:
- A [README](https://github.com/pingsutw/hadoop-submarine/tree/SUBMARINE-296/submarine-sdk/pysubmarine/example/deepfm) describes how to run DeepFM on locally and on submarine
- reorganize source code structure:
- Put all deepFM.json/run_deepFM.py and data into submarine-sdk/pysubmarine/examples/deepfm
- move ./submarine/ml/utils related code to ./submarine/utils/
- Fix code style, follow [Python Naming Conventions](https://visualgit.readthedocs.io/en/latest/pages/naming_convention.html)
- reorg JSON spec, move system parameter in `train` to `resource`. e.g. [deepfm.json](https://github.com/pingsutw/hadoop-submarine/blob/SUBMARINE-296/submarine-sdk/pysubmarine/example/deepfm/deepfm.json)
- support users to pass a list of input
- Add DeepFM Reference in comment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] wangdatan commented on issue #132: SUBMARINE-296. [SDK]
Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
wangdatan commented on issue #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#issuecomment-587111011
And I'm not quite sure how we run this with submarine CLI or REST API, it will be helpful if we can add a README.md to examples/deep_fm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #132:
SUBMARINE-296 [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #132: SUBMARINE-296 [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r361063224
##########
File path: submarine-sdk/pysubmarine/tests/ml/model/test_deepfm.py
##########
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from submarine.ml.model import deepFM
+import os
+
+LIBSVM_DATA = """1 1:0 2:0.051495 3:0.5 4:0.1 5:0.113437 6:0.874 7:0.01 8:0.08 9:0.028 10:0
+1 1:1.35 2:0.031561 3:0.45 4:0.56 5:0.000031 6:0.056 7:0.27 8:0.58 9:0.056 10:0.166667
+1 1:0.05 2:0.004983 3:0.19 4:0.14 5:0.000016 6:0.006 7:0.01 8:0.14 9:0.014 10:0.166667
+1 1:0.2 2:0.004983 3:0 4:0.12 5:0.016422 6:0.268 7:0.04 8:0.7 9:0.144 10:0.166667
+1 1:0 2:0.051495 3:0.5 4:0.1 5:0.113437 6:0.874 7:0.01 8:0.08 9:0.028 10:0
+1 1:1.35 2:0.031561 3:0.45 4:0.56 5:0.000031 6:0.056 7:0.27 8:0.58 9:0.056 10:0.166667
+1 1:0.05 2:0.004983 3:0.19 4:0.14 5:0.000016 6:0.006 7:0.01 8:0.14 9:0.014 10:0.166667
+1 1:0.2 2:0.004983 3:0 4:0.12 5:0.016422 6:0.268 7:0.04 8:0.7 9:0.144 10:0.166667
+"""
+
+
+def test_run_deepfm(tmpdir):
+ data_file = os.path.join(tmpdir, "libsvm.txt")
+ save_model_dir = os.path.join(tmpdir, "experiment")
+ with open(data_file, "wt") as writer:
+ writer.write(LIBSVM_DATA)
+
+ params = {
+ "input": {
+ "train_data": data_file,
+ "valid_data": data_file,
+ "test_data": data_file,
+ "type": "libsvm"
+ },
+ "output": {
+ "save_model_dir": save_model_dir,
+ "metric": "auc"
+ },
+ "training": {
+ "batch_size": 256,
+ "num_epochs": 1,
+ "field_size": 10,
+ "feature_size": 1000
+ }
+ }
+ model = deepFM(model_params=params)
+ model.train()
Review comment:
yes, by default we set `params['training']['mode']='local'` for local training, if we set to **distributed**, we use the submarine client to submit a job. Users don't need to modify any line of code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #132:
SUBMARINE-296 [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #132: SUBMARINE-296 [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r361044110
##########
File path: submarine-sdk/pysubmarine/tests/ml/model/test_deepfm.py
##########
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from submarine.ml.model import deepFM
+import os
+
+LIBSVM_DATA = """1 1:0 2:0.051495 3:0.5 4:0.1 5:0.113437 6:0.874 7:0.01 8:0.08 9:0.028 10:0
+1 1:1.35 2:0.031561 3:0.45 4:0.56 5:0.000031 6:0.056 7:0.27 8:0.58 9:0.056 10:0.166667
+1 1:0.05 2:0.004983 3:0.19 4:0.14 5:0.000016 6:0.006 7:0.01 8:0.14 9:0.014 10:0.166667
+1 1:0.2 2:0.004983 3:0 4:0.12 5:0.016422 6:0.268 7:0.04 8:0.7 9:0.144 10:0.166667
+1 1:0 2:0.051495 3:0.5 4:0.1 5:0.113437 6:0.874 7:0.01 8:0.08 9:0.028 10:0
+1 1:1.35 2:0.031561 3:0.45 4:0.56 5:0.000031 6:0.056 7:0.27 8:0.58 9:0.056 10:0.166667
+1 1:0.05 2:0.004983 3:0.19 4:0.14 5:0.000016 6:0.006 7:0.01 8:0.14 9:0.014 10:0.166667
+1 1:0.2 2:0.004983 3:0 4:0.12 5:0.016422 6:0.268 7:0.04 8:0.7 9:0.144 10:0.166667
+"""
+
+
+def test_run_deepfm(tmpdir):
+ data_file = os.path.join(tmpdir, "libsvm.txt")
+ save_model_dir = os.path.join(tmpdir, "experiment")
+ with open(data_file, "wt") as writer:
+ writer.write(LIBSVM_DATA)
+
+ params = {
+ "input": {
+ "train_data": data_file,
+ "valid_data": data_file,
+ "test_data": data_file,
+ "type": "libsvm"
+ },
+ "output": {
+ "save_model_dir": save_model_dir,
+ "metric": "auc"
+ },
+ "training": {
+ "batch_size": 256,
+ "num_epochs": 1,
+ "field_size": 10,
+ "feature_size": 1000
+ }
+ }
+ model = deepFM(model_params=params)
+ model.train()
Review comment:
What if I set the "model" to "distributed" here? I guess we will use the submarine client to submit a job spec to the cluster, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #132:
SUBMARINE-296 [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #132: SUBMARINE-296 [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r361062775
##########
File path: submarine-sdk/pysubmarine/example/deepFM.json
##########
@@ -0,0 +1,31 @@
+{
+ "input": {
+ "train_data": "./data/tr.libsvm",
Review comment:
@tangzhankun Thanks for the review. I like hyphen as well, but I follow other model convention like [xgboost](https://xgboost.readthedocs.io/en/latest/parameter.html), [sklearn](https://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html), it looks like data scientists much prefer to underscore
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] wangdatan commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
wangdatan commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r380318631
##########
File path: submarine-sdk/pysubmarine/submarine/ml/parameters.py
##########
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+LIBSVM = "libsvm"
Review comment:
Suggest to move this const to registries.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yanboliang commented on a change in pull request #132:
SUBMARINE-296. [SDK] Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
yanboliang commented on a change in pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132#discussion_r380489686
##########
File path: submarine-sdk/pysubmarine/example/deepFM.json
##########
@@ -0,0 +1,31 @@
+{
+ "input": {
+ "train_data": "./data/tr.libsvm",
Review comment:
qq: Have you ever check the workflow here? If users had installed submarine projects, HDFS should be there and used as the default file storage. Do the users need to upload these data to HDFS or we should pre-install them and put them on HDFS by default?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw closed pull request #132: SUBMARINE-296. [SDK]
Submarine pipeline example
Posted by GitBox <gi...@apache.org>.
pingsutw closed pull request #132: SUBMARINE-296. [SDK] Submarine pipeline example
URL: https://github.com/apache/submarine/pull/132
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org