You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@systemml.apache.org by mb...@apache.org on 2018/08/05 21:58:04 UTC

[1/4] systemml git commit: [SYSTEMML-445] Added builtin functions for efficient computation of lstm_backward function

Repository: systemml
Updated Branches:
  refs/heads/gh-pages f5ae0596d -> 0284f593f


[SYSTEMML-445] Added builtin functions for efficient computation of lstm_backward function

- The current implementation treats lstm and lstm_backward as stateless
  function for simplicity. We can revisit this after performance testing.
- Removed reserve parameter from lstm builtin function.
- Updated the language reference and lstm_staging.dml file.
- Added necessary kernels for transforming input to the format required by
  lstm_backward function.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/af4cf766
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/af4cf766
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/af4cf766

Branch: refs/heads/gh-pages
Commit: af4cf766f6e5e03d9ff1cdfe1a212bcdeb962d37
Parents: f5ae059
Author: Niketan Pansare <np...@us.ibm.com>
Authored: Fri Jun 15 13:07:59 2018 -0700
Committer: Niketan Pansare <np...@us.ibm.com>
Committed: Fri Jun 15 13:07:59 2018 -0700

----------------------------------------------------------------------
 dml-language-reference.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/systemml/blob/af4cf766/dml-language-reference.md
----------------------------------------------------------------------
diff --git a/dml-language-reference.md b/dml-language-reference.md
index 3212806..5bf9099 100644
--- a/dml-language-reference.md
+++ b/dml-language-reference.md
@@ -1520,7 +1520,7 @@ Hence, the images are internally represented as a matrix with dimension (N, C *
 | max_pool_backward, avg_pool_backward        | input, dout              | [batch_size X num_channels* height_image* width_image]    | [batch_size X num_channels* height_out* width_out]        | [batch_size X num_channels* height_image* width_image]                                      | stride=[stride_h, stride_w], padding=[pad_h, pad_w], input_shape=[batch_size, num_channels, height_image, width_image], pool_size=[height_pool, width_pool]                                   | Computes the gradients wrt input of 2D max pooling, average pooling                                                                               |
 | bias_add                                    | input, bias              | [batch_size X num_channels* height_image* width_image]    | [num_channels X 1]                                        | [batch_size X num_channels* height_image* width_image]                                      |                                                                                                                                                                                               | Adds the bias (row vector of size num_channels) to input with the given num_channels                                                              |
 | bias_multiply                               | input, bias              | [batch_size X num_channels* height_image* width_image]    | [num_channels X 1]                                        | [batch_size X num_channels* height_image* width_image]                                      |                                                                                                                                                                                               | Multiplies the bias (row vector of size num_channels) to input with the given num_channels                                                        |
-| lstm                                        | X,  W, bias, out0, c0    | [batch_size X seq_length*num_features]                    | [num_features+hidden_size X 4*hidden_size]                | [batch_size X seq_length*hidden_size] if return_sequences else  [batch_size X hidden_size]  | return_sequences                                                                                                                                                                              | Perform computation for single-layer unidirectional LSTM (outputs: out, carryOut, reserveSpace)                                                   |
+| lstm                                        | X,  W, bias, out0, c0    | [batch_size X seq_length*num_features]                    | [num_features+hidden_size X 4*hidden_size]                | [batch_size X seq_length*hidden_size] if return_sequences else  [batch_size X hidden_size]  | return_sequences                                                                                                                                                                              | Perform computation for single-layer unidirectional LSTM (outputs: out, carryOut)                                                                 |
 | batch_norm2d                                | input                    | [batch_size X num_channels* height_image* width_image]    |                                                           | [batch_size X num_channels* height_image* width_image]                                      | scale, shift, exponentialMovingAverage_Mean, exponentialMovingAverage_Variance, mode, epsilon, momentum                                                                                       | Performs batch normalization operation  (outputs: updated exponential moving average mean and variance, cache of the batch mean and variance)     |
 | batch_norm2d_backward                       | input, dout              | [batch_size X num_channels* height_image* width_image]    | [batch_size X num_channels* height_image* width_image]    | [batch_size X num_channels* height_image* width_image]                                      | scale, epsilon, cache_mean (from forward), cache_inv_var (from forward)                                                                                                                       | Computed backpropagation error for batch normalization operation                                                                                  |
 


[2/4] systemml git commit: [MINOR] Profile memory use in JMLC execution

Posted by mb...@apache.org.
[MINOR] Profile memory use in JMLC execution

This PR adds utilities to profile memory use during execution in JMLC. Specifically, the following changes were made:

1. Added options setStatistics() and gatherMemStats() to api.jmlc.Connection which control whether or not statistics should be gathered, and if so, whether memory use should be profiled. Also added an appropriate method to api.jmlc.PreparedScript to display the resulting statistics. The following points are only applicable when running in JMLC mode, and memory statistics have been enabled. Both these options are false by default.
2. Modified utils.Statistics to track the memory used by distinct CacheBlock objects. At the conclusion of the script, the maximum memory use is reported. Memory use is computed by calling the object's getInMemorySize() method. This will generally be a slight over-estimate of the actual memory used by the object.
3. If FINEGRAINED_STATISTICS are enabled, Statistics will also track the memory use by each named variable in a DML script and report this in a table as in heavy hitter instructions. The goal of this is to detect unexpected large intermediate matrices (e.g. resulting from an outer product X %*% t(X)).
4. If FINEGRAINED_STATISTICS are enabled, Statistics will attempt to measure more accurate memory use by checking to see if an object has been garbage collected. This is done by maintaining a soft reference to the object and periodically checking to see if it has become null. This is enabled only when using fine-grained statistics since it introduces potentially non-trivial overheads by scanning a list of live objects. Note that simply using rmvar to remove a live variable results in a substantial underestimate of memory used by the program and so this method is not used. When finegrained statistics are not enabled, the resulting statistics will be an overestimate.

Potential impacts to performance: when finegrained statistics are enabled there will be some performance degradation from maintaining the set of live variables.

Potential Improvements: Related to the above, it would be nice to find a way of accurately tracking when an object is actually released without resorting to checking whether a soft reference has become null. It might also be nice to include a line number indicating where a "heavy hitting object" was created to make debugging easier.

Closes #794.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/a2d3a721
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/a2d3a721
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/a2d3a721

Branch: refs/heads/gh-pages
Commit: a2d3a721d05851ed03cb3fa5320075d7872b7ed0
Parents: af4cf76
Author: Anthony Thomas <ah...@eng.ucsd.edu>
Authored: Fri Jul 6 11:10:17 2018 -0700
Committer: Niketan Pansare <np...@us.ibm.com>
Committed: Fri Jul 6 11:23:36 2018 -0700

----------------------------------------------------------------------
 jmlc.md | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/systemml/blob/a2d3a721/jmlc.md
----------------------------------------------------------------------
diff --git a/jmlc.md b/jmlc.md
index 2183700..a703d01 100644
--- a/jmlc.md
+++ b/jmlc.md
@@ -49,6 +49,18 @@ of SystemML's distributed modes, such as Spark batch mode or Hadoop batch mode,
 distributed computing capabilities. JMLC offers embeddability at the cost of performance, so its use is
 dependent on the nature of the business use case being addressed.
 
+## Statistics
+
+JMLC can be configured to gather runtime statistics, as in the MLContext API, by calling Connection's `setStatistics()`
+method with a value of `true`. JMLC can also be configured to gather statistics on the memory used by matrices and
+frames in the DML script. To enable collection of memory statistics, call Connection's `gatherMemStats()` method
+with a value of `true`. When finegrained statistics are enabled in `SystemML.conf`, JMLC will also report the variables
+in the DML script which used the most memory. By default, the memory use reported will be an overestimte of the actual
+memory required to run the program. When finegrained statistics are enabled, JMLC will gather more accurate statistics
+by keeping track of garbage collection events and reducing the memory estimate accordingly. The most accurate way to
+determine the memory required by a script is to run the script in a single thread and enable finegrained statistics.
+
+An example showing how to enable statistics in JMLC is presented in the section below.
 
 ---
 
@@ -114,11 +126,19 @@ the resulting `"predicted_y"` matrix. We repeat this process. When done, we clos
  
         // obtain connection to SystemML
         Connection conn = new Connection();
+
+        // turn on gathering of runtime statistics and memory use
+        conn.setStatistics(true);
+        conn.gatherMemStats(true);
  
         // read in and precompile DML script, registering inputs and outputs
         String dml = conn.readScript("scoring-example.dml");
         PreparedScript script = conn.prepareScript(dml, new String[] { "W", "X" }, new String[] { "predicted_y" }, false);
- 
+
+        // obtain the runtime plan generated by SystemML
+        String plan = script.explain();
+        System.out.println(plan);
+
         double[][] mtx = matrix(4, 3, new double[] { 1, 2, 3, 4, 5, 6, 7, 8, 9 });
         double[][] result = null;
  
@@ -127,6 +147,10 @@ the resulting `"predicted_y"` matrix. We repeat this process. When done, we clos
         script.setMatrix("X", randomMatrix(3, 3, -1, 1, 0.7));
         result = script.executeScript().getMatrix("predicted_y");
         displayMatrix(result);
+
+        // print the resulting runtime statistics
+        String stats = script.statistics();
+        System.out.println(stats);
  
         script.setMatrix("W", mtx);
         script.setMatrix("X", randomMatrix(3, 3, -1, 1, 0.7));


[3/4] systemml git commit: [MINOR] Improved JMLC memory profiling, command line arg -mem

Posted by mb...@apache.org.
[MINOR] Improved JMLC memory profiling, command line arg -mem

Closes #797.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/9db274fe
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/9db274fe
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/9db274fe

Branch: refs/heads/gh-pages
Commit: 9db274fe67bfbee5b348bd8db5274621d718e9b5
Parents: a2d3a72
Author: Anthony Thomas <ah...@ibm.com>
Authored: Thu Jul 12 21:30:22 2018 -0700
Committer: Matthias Boehm <mb...@gmail.com>
Committed: Thu Jul 12 21:30:23 2018 -0700

----------------------------------------------------------------------
 jmlc.md | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/systemml/blob/9db274fe/jmlc.md
----------------------------------------------------------------------
diff --git a/jmlc.md b/jmlc.md
index a703d01..08d1688 100644
--- a/jmlc.md
+++ b/jmlc.md
@@ -55,12 +55,8 @@ JMLC can be configured to gather runtime statistics, as in the MLContext API, by
 method with a value of `true`. JMLC can also be configured to gather statistics on the memory used by matrices and
 frames in the DML script. To enable collection of memory statistics, call Connection's `gatherMemStats()` method
 with a value of `true`. When finegrained statistics are enabled in `SystemML.conf`, JMLC will also report the variables
-in the DML script which used the most memory. By default, the memory use reported will be an overestimte of the actual
-memory required to run the program. When finegrained statistics are enabled, JMLC will gather more accurate statistics
-by keeping track of garbage collection events and reducing the memory estimate accordingly. The most accurate way to
-determine the memory required by a script is to run the script in a single thread and enable finegrained statistics.
-
-An example showing how to enable statistics in JMLC is presented in the section below.
+in the DML script which used the most memory. An example showing how to enable statistics in JMLC is presented in the
+section below.
 
 ---
 


[4/4] systemml git commit: [SYSTEMML-2090] Language documentation for paramserv builtin function

Posted by mb...@apache.org.
[SYSTEMML-2090] Language documentation for paramserv builtin function

Closes #816.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/0284f593
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/0284f593
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/0284f593

Branch: refs/heads/gh-pages
Commit: 0284f593fa660ea81a98c8735cef346164391c98
Parents: 9db274f
Author: EdgarLGB <gu...@atos.net>
Authored: Sun Aug 5 14:37:59 2018 -0700
Committer: Matthias Boehm <mb...@gmail.com>
Committed: Sun Aug 5 14:53:50 2018 -0700

----------------------------------------------------------------------
 dml-language-reference.md | 77 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/systemml/blob/0284f593/dml-language-reference.md
----------------------------------------------------------------------
diff --git a/dml-language-reference.md b/dml-language-reference.md
index 5bf9099..8a89d99 100644
--- a/dml-language-reference.md
+++ b/dml-language-reference.md
@@ -53,6 +53,7 @@ limitations under the License.
     * [Read/Write Built-In Functions](dml-language-reference.html#readwrite-built-in-functions)
     * [Data Pre-Processing Built-In Functions](dml-language-reference.html#data-pre-processing-built-in-functions)
     * [Deep Learning Built-In Functions](dml-language-reference.html#deep-learning-built-in-functions)
+    * [Parameter Server Built-In Function](dml-language-reference.html#parameter-server-built-in-function)
     * [Other Built-In Functions](dml-language-reference.html#other-built-in-functions)
   * [Frames](dml-language-reference.html#frames)
     * [Creating Frames](dml-language-reference.html#creating-frames)
@@ -1536,6 +1537,82 @@ Examples:
 | bias_add             |                             | `ones = matrix(1, rows=1, cols=height*width); output = input + matrix(bias %*% ones, rows=1, cols=numChannels*height*width)`                                |
 | bias_multiply        |                             | `ones = matrix(1, rows=1, cols=height*width); output = input * matrix(bias %*% ones, rows=1, cols=numChannels*height*width)`                                |
 
+### Parameter Server Built-in Function
+Apart from data-parallel operations and task-parallel parfor loops, SystemML also supports a **data-parallel Parameter Server** via a built-in function **paramserv**. Currently both local multi-threaded and spark distributed backend are supported to execute the **paramserv** function. So far we only support a single parameter server with N workers as well as synchronous and asynchronous model updates per batch or epoch. For example, in order to train a model in local backend with update strategy BSP, 10 epochs, 64 batchsize, 10 workers, **paramserv** function should look like this:
+
+
+    resultModel=paramserv(model=initModel, features=X, labels=Y, 
+                          upd="fun1", agg="fun2", epochs=10, k=10, hyperparams=hParams)
+
+
+**Table**: Inputs of paramserv function
+
+Parameters | Description | Type | Mandatory | Options
+-------- | ----------- | ---------- | ---------- | -------
+model | All the parameters (e.g., the weight and bias matrices) | list | yes | 
+features | Training features | matrix | yes 
+labels | Training labels | matrix | yes
+val_features | Validation features | matrix | no
+val_labels | Validation labels | matrix | no
+upd | Physical name of gradient calculation function. The format should be "related path:func name". For example, "./mnist_lenet_paramserv_sgd.dml::gradients". | string | yes
+agg | Physical name of gradient aggregation function. The format should be "related path:func name". For example, "./mnist_lenet_paramserv_sgd.dml::aggregation". | string | yes
+mode | Execution backend for data partitioning and worker execution | string | no | "LOCAL"(default), "REMOTE_SPARK"
+utype | Update strategy | string | no | "ASP"(default), "BSP"
+freq | Frequency of model updating | string | no | "EPOCH"(default), "BATCH"
+epochs | Number of epochs, where an epoch is a full scan over the data | integer | yes |
+batchsize | Size of a mini-batch (number of rows) | integer | no | 64(default)
+k | Number of workers | integer | no | Number of vcores(default)
+scheme | Scheme of data partition, i.e., how the data is distributed across workers | string | no | "DISJOINT_CONTIGUOUS"(default), "DISJOINT_ROUND_ROBIN", "DISJOINT_RANDOM", "OVERLAP_RESHUFFLE"
+hyperparams | Additional hyper parameters, e.g., learning rate, momentum | list | yes | 
+checkpointing | Checkpoint strategy, currently not supported | string | no | 
+
+**Table**: Output of paramserv function
+
+Output | Description | Type
+-------- | ----------- | ----------
+model | Trained model | list
+
+**Update function:**
+
+The update function calculates the gradients for a single mini-batch and the given model (e.g., via a forward and backward pass through a neural network). The implementation of this function should be based on a function signature like this: (i.e., **the input parameter including both type and name should be exactly the same as the below, except that the output name could be different**)
+
+```sh
+gradients = function(list[unknown] model, list[unknown] hyperparams,
+                     matrix[double] features, matrix[double] labels)
+          return (list[unknown] gradients)
+          # the output name can be something else than "gradients" but should always return a list
+```
+
+**Aggregate function:**
+
+The aggregate function then takes the computed or accrued gradients and updates the model via some optimizer such as Adagrad or Adam. The implementation of this function should be based on a function signature like this: (i.e., **the input parameter including both type and name should be exactly the same as the below, except that the output name could be different**)
+
+```sh
+aggregation = function(list[unknown] model, list[unknown] hyperparams,
+                       list[unknown] gradients)
+         return (list[unknown] modelResult)
+         # the output name can be something else than "modelResult" but should always return a list
+```
+
+**Update strategy:**
+
+Currently, two types of update strategy, **ASP** and **BSP**, are supported. **ASP**, a.k.a. _Asynchronous Parallel_, means that the model updates will be completed in an asynchronous manner. The parameter server updates the model and broadcasts the updated model immediately with the fresh gradients pushed by the worker and then the worker is able to pull the new updated model. This push-and-pull process is done asynchronously across workers. While **BSP**, a.k.a. _Bulk Synchronous Parallel_, the server will update the global model until having received all the gradients sent by workers in one iteration and then workers could move into the next iteration. Hence, the overall performance is affected by stragglers (i.e., the slowest worker).
+
+**Update frequency:**
+
+When pushing the gradients from workers to server for updating the model, we could determine how often this push-and-pull process will be done. Currently, two types of update frequency, **EPOCH** and **BATCH** are supported. When setting to **EPOCH**, the generated gradients of each mini-batch are accumulated locally in each worker. The accrued gradients are then pushed to the server whenever a worker finished an epoch. While setting to **BATCH**, the generated gradients of each mini-batch are pushed to server immediately to launch the push-and-pull process.
+
+**Data partition schemes:**
+
+Before launching the data-parallel parameter server, the original data will be partitioned across workers according to some schemes. Currently, four types of schemes are supported, Disjoint_Contigous, Disjoint_Round_Robin, Disjoint_Random, Overlap_Reshuffle.
+
+Scheme | Definition
+-------- | -----------
+Disjoint_Contiguous | For each worker, use a right indexing operation X[beg:end,] to obtain contiguous, non-overlapping partitions of rows
+Disjoint_Round_Robin | For each worker, use a permutation multiply or simpler a removeEmpty such as removeEmpty(target=X, margin=rows, select=(seq(1,nrow(X))%%k)==id)
+Disjoint_Random | For each worker, use a permutation multiply P[beg:end,] %*% X, where P is constructed for example with P=table(seq(1,nrow(X),sample(nrow(X), nrow(X)))), i.e., sampling without replacement to ensure disjointness
+Overlap_Reshuffle | Similar to the above, except to create a new permutation matrix for each worker and without the indexing on P
+
 ### Other Built-In Functions
 
 **Table 16**: Other Built-In Functions