You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@systemds.apache.org by ba...@apache.org on 2022/11/15 14:49:06 UTC
[systemds] branch main updated (2ed900494b -> dae9981476)
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
from 2ed900494b [DOCS] Update docs
new 15e278a257 [SYSTEMDS-3464] Python Combine Write
new dae9981476 [DOCS] Update Python docs
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/api/java/member-search-index.zip | Bin 229274 -> 229274 bytes
docs/api/java/package-search-index.zip | Bin 869 -> 869 bytes
docs/api/java/type-search-index.zip | Bin 14277 -> 14277 bytes
docs/api/python/api/context/systemds_context.html | 15 ++++
docs/api/python/genindex.html | 2 +
docs/api/python/guide/python_end_to_end_tut.html | 85 ++++++++++++++++-----
docs/api/python/objects.inv | Bin 2298 -> 2302 bytes
docs/api/python/searchindex.js | 2 +-
.../sources/guide/python_end_to_end_tut.rst.txt | 34 ++++++---
.../docs/source/code/guide/end_to_end/part2.py | 26 +++++--
.../docs/source/guide/python_end_to_end_tut.rst | 34 ++++++---
.../python/systemds/context/systemds_context.py | 15 +++-
src/main/python/systemds/operator/__init__.py | 6 +-
.../{algorithm/builtin/WoE.py => nodes/combine.py} | 56 ++++++--------
14 files changed, 192 insertions(+), 83 deletions(-)
copy src/main/python/systemds/operator/{algorithm/builtin/WoE.py => nodes/combine.py} (52%)
[systemds] 01/02: [SYSTEMDS-3464] Python Combine Write
Posted by ba...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
commit 15e278a257ead5cf42891d2970664ba5f0682e80
Author: baunsgaard <ba...@tugraz.at>
AuthorDate: Tue Nov 15 15:19:06 2022 +0100
[SYSTEMDS-3464] Python Combine Write
This commit finally adds a long wanted feature to python scripts.
This allows us to call multiple end nodes in a script without having
to do multiple executions.
Combine(Write(X,"Path1"), Write(Y,"Path2")).compute()
Similarly we can do:
Combine(Print(X), Write(X,"Path1)).compute()
Closes #1729
---
.../docs/source/code/guide/end_to_end/part2.py | 26 ++++++++---
.../docs/source/guide/python_end_to_end_tut.rst | 34 +++++++++-----
.../python/systemds/context/systemds_context.py | 15 ++++++-
src/main/python/systemds/operator/__init__.py | 6 ++-
src/main/python/systemds/operator/nodes/combine.py | 52 ++++++++++++++++++++++
5 files changed, 114 insertions(+), 19 deletions(-)
diff --git a/src/main/python/docs/source/code/guide/end_to_end/part2.py b/src/main/python/docs/source/code/guide/end_to_end/part2.py
index 73c7a6539f..a408e2796a 100644
--- a/src/main/python/docs/source/code/guide/end_to_end/part2.py
+++ b/src/main/python/docs/source/code/guide/end_to_end/part2.py
@@ -40,15 +40,11 @@ with SystemDSContext() as sds:
# Transform frames to matrices.
X, M1 = X_frame.transform_encode(spec=jspec_data)
- Xt = Xt_frame.transform_apply(spec=jspec_data, meta=M1)
Y, M2 = Y_frame.transform_encode(spec=jspec_labels)
- Yt = Yt_frame.transform_apply(spec=jspec_labels, meta=M2)
# Subsample to make training faster
X = X[0:train_count]
Y = Y[0:train_count]
- Xt = Xt[0:test_count]
- Yt = Yt[0:test_count]
# Load custom neural network
neural_net_src_path = "tests/examples/tutorials/neural_net_source.dml"
@@ -60,6 +56,26 @@ with SystemDSContext() as sds:
seed = 42
network = FFN_package.train(X, Y, epochs, batch_size, learning_rate, seed)
+
+ # Write metadata and trained network to disk.
+ sds.combine(
+ network.write('tests/examples/docs_test/end_to_end/network'),
+ M1.write('tests/examples/docs_test/end_to_end/encode_X'),
+ M2.write('tests/examples/docs_test/end_to_end/encode_Y')
+ ).compute()
- network.write('tests/examples/docs_test/end_to_end/').compute()
+ # Read metadata and trained network and do prediction.
+ M1_r = sds.read('tests/examples/docs_test/end_to_end/encode_X')
+ M2_r = sds.read('tests/examples/docs_test/end_to_end/encode_Y')
+ network_r = sds.read('tests/examples/docs_test/end_to_end/network')
+ Xt = Xt_frame.transform_apply(spec=jspec_data, meta=M1_r)
+ Yt = Yt_frame.transform_apply(spec=jspec_labels, meta=M2_r)
+ Xt = Xt[0:test_count]
+ Yt = Yt[0:test_count]
+ FFN_package_2 = sds.source(neural_net_src_path, "fnn")
+ probs = FFN_package_2.predict(Xt, network_r)
+ accuracy = FFN_package_2.eval(probs, Yt).compute()
+ import logging
+ logging.info("accuracy: " + str(accuracy))
+
diff --git a/src/main/python/docs/source/guide/python_end_to_end_tut.rst b/src/main/python/docs/source/guide/python_end_to_end_tut.rst
index 2a8cb9fb76..961b47d61b 100644
--- a/src/main/python/docs/source/guide/python_end_to_end_tut.rst
+++ b/src/main/python/docs/source/guide/python_end_to_end_tut.rst
@@ -118,12 +118,13 @@ For this we will introduce another dml file, which can be used to train a basic
Step 1: Obtain data
~~~~~~~~~~~~~~~~~~~
-For the whole data setup please refer to level 1, Step 1, as these steps are identical.
+For the whole data setup please refer to level 1, Step 1, as these steps are almost identical,
+but instead of preparing the test data, we only prepare the training data.
.. include:: ../code/guide/end_to_end/part2.py
:code: python
:start-line: 20
- :end-line: 51
+ :end-line: 47
Step 2: Load the algorithm
~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -134,12 +135,10 @@ This file includes all the necessary functions for training, evaluating, and sto
The returned object of the source call is further used for calling the functions.
The file can be found here:
- - :doc:tests/examples/tutorials/neural_net_source.dml
-
.. include:: ../code/guide/end_to_end/part2.py
:code: python
- :start-line: 54
- :end-line: 55
+ :start-line: 48
+ :end-line: 51
Step 3: Training the neural network
@@ -153,8 +152,8 @@ The seed argument ensures that running the code again yields the same results.
.. include:: ../code/guide/end_to_end/part2.py
:code: python
- :start-line: 61
- :end-line: 62
+ :start-line: 52
+ :end-line: 58
Step 4: Saving the model
@@ -163,15 +162,28 @@ Step 4: Saving the model
For later usage, we can save the trained model.
We only need to specify the name of our model and the file path.
This call stores the weights and biases of our model.
+Similarly the transformation metadata to transform input data to the model,
+is saved.
.. include:: ../code/guide/end_to_end/part2.py
:code: python
- :start-line: 64
+ :start-line: 59
:end-line: 65
+Step 5: Predict on Unseen data
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once the model is saved along with metadata, it is simple to apply it all to
+unseen data:
+
+.. include:: ../code/guide/end_to_end/part2.py
+ :code: python
+ :start-line: 66
+ :end-line: 77
+
Full Script NN
-~~~~~~~~~~~---
+~~~~~~~~~~~~~~
The complete script now can be seen here:
@@ -179,4 +191,4 @@ The complete script now can be seen here:
.. include:: ../code/guide/end_to_end/part2.py
:code: python
:start-line: 20
- :end-line: 64
+ :end-line: 80
diff --git a/src/main/python/systemds/context/systemds_context.py b/src/main/python/systemds/context/systemds_context.py
index 8543e83417..a4c89d557e 100644
--- a/src/main/python/systemds/context/systemds_context.py
+++ b/src/main/python/systemds/context/systemds_context.py
@@ -38,7 +38,7 @@ import numpy as np
import pandas as pd
from py4j.java_gateway import GatewayParameters, JavaGateway, Py4JNetworkError
from systemds.operator import (Frame, List, Matrix, OperationNode, Scalar,
- Source)
+ Source, Combine)
from systemds.script_building import DMLScript, OutputType
from systemds.utils.consts import VALID_INPUT_TYPES
from systemds.utils.helpers import get_module_dir
@@ -630,6 +630,19 @@ class SystemDSContext(object):
"""
return List(self, unnamed_input_nodes=args, named_input_nodes=kwargs)
+ def combine(self, *args: Sequence[VALID_INPUT_TYPES]) -> Combine:
+ """ combine nodes to call compute on multiple operations.
+
+ This is usefull for the case of having multiple writes in one script and wanting
+ to execute all in one execution reusing intermediates.
+
+ Note this combine does not allow to return anything to the user, so if used,
+ please only use nodes that end with either writing or printing elements.
+
+ :param args: A sequence that will be executed with call to compute()
+ """
+ return Combine(self, unnamed_input_nodes=args)
+
def array(self, *args: Sequence[VALID_INPUT_TYPES]) -> List:
""" Create a List object containing the given nodes.
diff --git a/src/main/python/systemds/operator/__init__.py b/src/main/python/systemds/operator/__init__.py
index c1577441c8..51a586bc63 100644
--- a/src/main/python/systemds/operator/__init__.py
+++ b/src/main/python/systemds/operator/__init__.py
@@ -20,13 +20,15 @@
# -------------------------------------------------------------
from systemds.operator.operation_node import OperationNode
-from systemds.operator.nodes.multi_return import MultiReturn
from systemds.operator.nodes.scalar import Scalar
from systemds.operator.nodes.matrix import Matrix
+from systemds.operator.nodes.multi_return import MultiReturn
from systemds.operator.nodes.frame import Frame
+from systemds.operator.nodes.combine import Combine
from systemds.operator.nodes.list_access import ListAccess
from systemds.operator.nodes.list import List
from systemds.operator.nodes.source import Source
from systemds.operator import algorithm
-__all__ = ["OperationNode", "algorithm", "Scalar", "List", "ListAccess", "Matrix", "Frame", "Source", "MultiReturn"]
+__all__ = ["OperationNode", "algorithm", "Scalar", "List",
+ "ListAccess", "Matrix", "Frame", "Source", "MultiReturn", "Combine"]
diff --git a/src/main/python/systemds/operator/nodes/combine.py b/src/main/python/systemds/operator/nodes/combine.py
new file mode 100644
index 0000000000..9f7e9f0439
--- /dev/null
+++ b/src/main/python/systemds/operator/nodes/combine.py
@@ -0,0 +1,52 @@
+# -------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# -------------------------------------------------------------
+
+
+__all__ = ["Combine"]
+
+from typing import Dict, Iterable, List, Sequence
+
+from systemds.operator import OperationNode
+from systemds.script_building.dag import OutputType
+from systemds.utils.consts import VALID_INPUT_TYPES
+
+
+class Combine(OperationNode):
+
+ def __init__(self, sds_context, func='',
+ unnamed_input_nodes: Iterable[OperationNode] = None):
+ for a in unnamed_input_nodes:
+ if(a.output_type != OutputType.NONE):
+ raise ValueError(
+ "Cannot combine elements that have outputs, all elements must be instances of print or write")
+
+ self._outputs = {}
+ super().__init__(sds_context, func, unnamed_input_nodes, None, OutputType.NONE, False)
+
+ def code_line(self, var_name: str, unnamed_input_vars: Sequence[str],
+ named_input_vars: Dict[str, str]) -> str:
+ return ''
+
+ def compute(self, verbose: bool = False, lineage: bool = False):
+ return super().compute(verbose, lineage)
+
+ def __str__(self):
+ return "Combine"
[systemds] 02/02: [DOCS] Update Python docs
Posted by ba...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
commit dae998147668455c2fc67218df4fca6418689e15
Author: baunsgaard <ba...@tugraz.at>
AuthorDate: Tue Nov 15 15:48:37 2022 +0100
[DOCS] Update Python docs
---
docs/api/java/member-search-index.zip | Bin 229274 -> 229274 bytes
docs/api/java/package-search-index.zip | Bin 869 -> 869 bytes
docs/api/java/type-search-index.zip | Bin 14277 -> 14277 bytes
docs/api/python/api/context/systemds_context.html | 15 ++++
docs/api/python/genindex.html | 2 +
docs/api/python/guide/python_end_to_end_tut.html | 85 ++++++++++++++++-----
docs/api/python/objects.inv | Bin 2298 -> 2302 bytes
docs/api/python/searchindex.js | 2 +-
.../sources/guide/python_end_to_end_tut.rst.txt | 34 ++++++---
9 files changed, 105 insertions(+), 33 deletions(-)
diff --git a/docs/api/java/member-search-index.zip b/docs/api/java/member-search-index.zip
index 7d505cd54d..5ba0ae427a 100644
Binary files a/docs/api/java/member-search-index.zip and b/docs/api/java/member-search-index.zip differ
diff --git a/docs/api/java/package-search-index.zip b/docs/api/java/package-search-index.zip
index cfa4fe2264..04428621db 100644
Binary files a/docs/api/java/package-search-index.zip and b/docs/api/java/package-search-index.zip differ
diff --git a/docs/api/java/type-search-index.zip b/docs/api/java/type-search-index.zip
index 7f59f1fc64..6b2c59a496 100644
Binary files a/docs/api/java/type-search-index.zip and b/docs/api/java/type-search-index.zip differ
diff --git a/docs/api/python/api/context/systemds_context.html b/docs/api/python/api/context/systemds_context.html
index 0f546c3563..2050d7ba31 100644
--- a/docs/api/python/api/context/systemds_context.html
+++ b/docs/api/python/api/context/systemds_context.html
@@ -278,6 +278,21 @@ Afterwards capturing will be reset to the state it was before.</p>
<dd><p>Close the connection to the java process and do necessary cleanup.</p>
</dd></dl>
+<dl class="py method">
+<dt id="systemds.context.SystemDSContext.combine">
+<code class="sig-name descname">combine</code><span class="sig-paren">(</span><em class="sig-param"><span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Sequence<span class="p">[</span>Union<span class="p">[</span>DAGNode<span class="p">, </span>str<span class="p">, </span>int<span class="p">, </span>float<span class="p">, </span>bool<span class="p">]</span><span class="p">]</span></span></em><span class="sig-paren">)</span> → systemds.operat [...]
+<dd><p>combine nodes to call compute on multiple operations.</p>
+<p>This is usefull for the case of having multiple writes in one script and wanting
+to execute all in one execution reusing intermediates.</p>
+<p>Note this combine does not allow to return anything to the user, so if used,
+please only use nodes that end with either writing or printing elements.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><p><strong>args</strong> – A sequence that will be executed with call to compute()</p>
+</dd>
+</dl>
+</dd></dl>
+
<dl class="py method">
<dt id="systemds.context.SystemDSContext.dict">
<code class="sig-name descname">dict</code><span class="sig-paren">(</span><em class="sig-param"><span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Dict<span class="p">[</span>str<span class="p">, </span>Union<span class="p">[</span>DAGNode<span class="p">, </span>str<span class="p">, </span>int<span class="p">, </span>float<span class="p">, </span>bool<span class="p">]</span><span class="p">]</span></span></em><span class="sig-paren">)</span> [...]
diff --git a/docs/api/python/genindex.html b/docs/api/python/genindex.html
index 2bf0e31100..fd1b423e51 100644
--- a/docs/api/python/genindex.html
+++ b/docs/api/python/genindex.html
@@ -327,6 +327,8 @@
<li><a href="api/script_building/dag.html#systemds.script_building.dag.DAGNode.code_line">(systemds.script_building.dag.DAGNode method)</a>
</li>
</ul></li>
+ <li><a href="api/context/systemds_context.html#systemds.context.SystemDSContext.combine">combine() (systemds.context.SystemDSContext method)</a>
+</li>
<li><a href="api/operator/algorithms.html#systemds.operator.algorithm.components">components() (in module systemds.operator.algorithm)</a>
</li>
<li><a href="api/operator/node/frame.html#systemds.operator.Frame.compute">compute() (systemds.operator.Frame method)</a>
diff --git a/docs/api/python/guide/python_end_to_end_tut.html b/docs/api/python/guide/python_end_to_end_tut.html
index 9e17567270..634eb9b479 100644
--- a/docs/api/python/guide/python_end_to_end_tut.html
+++ b/docs/api/python/guide/python_end_to_end_tut.html
@@ -104,6 +104,8 @@
<li class="toctree-l3"><a class="reference internal" href="#step-2-load-the-algorithm">Step 2: Load the algorithm</a></li>
<li class="toctree-l3"><a class="reference internal" href="#step-3-training-the-neural-network">Step 3: Training the neural network</a></li>
<li class="toctree-l3"><a class="reference internal" href="#step-4-saving-the-model">Step 4: Saving the model</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#step-5-predict-on-unseen-data">Step 5: Predict on Unseen data</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#full-script-nn">Full Script NN</a></li>
</ul>
</li>
</ul>
@@ -342,7 +344,8 @@ for the predictions and the confusion matrix averages of each true class.</p>
For this we will introduce another dml file, which can be used to train a basic feed forward network.</p>
<div class="section" id="step-1-obtain-data">
<h3>Step 1: Obtain data<a class="headerlink" href="#step-1-obtain-data" title="Permalink to this headline">¶</a></h3>
-<p>For the whole data setup please refer to level 1, Step 1, as these steps are identical.</p>
+<p>For the whole data setup please refer to level 1, Step 1, as these steps are almost identical,
+but instead of preparing the test data, we only prepare the training data.</p>
<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">systemds.context</span> <span class="kn">import</span> <span class="n">SystemDSContext</span>
<span class="kn">from</span> <span class="nn">systemds.examples.tutorials.adult</span> <span class="kn">import</span> <span class="n">DataManager</span>
<span class="kn">from</span> <span class="nn">systemds.operator.algorithm</span> <span class="kn">import</span> <span class="n">multiLogReg</span>
@@ -365,15 +368,11 @@ For this we will introduce another dml file, which can be used to train a basic
<span class="c1"># Transform frames to matrices.</span>
<span class="n">X</span><span class="p">,</span> <span class="n">M1</span> <span class="o">=</span> <span class="n">X_frame</span><span class="o">.</span><span class="n">transform_encode</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_data</span><span class="p">)</span>
- <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_data</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M1</span><span class="p">)</span>
<span class="n">Y</span><span class="p">,</span> <span class="n">M2</span> <span class="o">=</span> <span class="n">Y_frame</span><span class="o">.</span><span class="n">transform_encode</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_labels</span><span class="p">)</span>
- <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_labels</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M2</span><span class="p">)</span>
<span class="c1"># Subsample to make training faster</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">X</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">train_count</span><span class="p">]</span>
<span class="n">Y</span> <span class="o">=</span> <span class="n">Y</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">train_count</span><span class="p">]</span>
- <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span>
- <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span>
</pre></div>
</div>
</div>
@@ -384,12 +383,9 @@ First, we need to source the dml file for neural networks.
This file includes all the necessary functions for training, evaluating, and storing the model.
The returned object of the source call is further used for calling the functions.
The file can be found here:</p>
-<blockquote>
-<div><ul class="simple">
-<li><p>:doc:tests/examples/tutorials/neural_net_source.dml</p></li>
-</ul>
-</div></blockquote>
-<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="n">FFN_package</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">source</span><span class="p">(</span><span class="n">neural_net_src_path</span><span class="p">,</span> <span class="s2">"fnn"</span><span class="p">)</span>
+<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="c1"># Load custom neural network</span>
+ <span class="n">neural_net_src_path</span> <span class="o">=</span> <span class="s2">"tests/examples/tutorials/neural_net_source.dml"</span>
+ <span class="n">FFN_package</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">source</span><span class="p">(</span><span class="n">neural_net_src_path</span><span class="p">,</span> <span class="s2">"fnn"</span><span class="p">)</span>
</pre></div>
</div>
</div>
@@ -400,7 +396,12 @@ The first two arguments are the training features and the target values we want
Then we need to set the hyperparameters of the model.
We choose to train for 1 epoch with a batch size of 16 and a learning rate of 0.01, which are common parameters for neural networks.
The seed argument ensures that running the code again yields the same results.</p>
-<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="n">network</span> <span class="o">=</span> <span class="n">FFN_package</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Y</span><span class="p">,</span> <span class="n">epochs</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">,</span> <span class="n"> [...]
+<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="n">epochs</span> <span class="o">=</span> <span class="mi">1</span>
+ <span class="n">batch_size</span> <span class="o">=</span> <span class="mi">16</span>
+ <span class="n">learning_rate</span> <span class="o">=</span> <span class="mf">0.01</span>
+ <span class="n">seed</span> <span class="o">=</span> <span class="mi">42</span>
+
+ <span class="n">network</span> <span class="o">=</span> <span class="n">FFN_package</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Y</span><span class="p">,</span> <span class="n">epochs</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">,</span> <span class="n">learning_rate</span><span class="p">,</span> <span class="n">seed</span><span class="p">)</span>
</pre></div>
</div>
</div>
@@ -408,12 +409,38 @@ The seed argument ensures that running the code again yields the same results.</
<h3>Step 4: Saving the model<a class="headerlink" href="#step-4-saving-the-model" title="Permalink to this headline">¶</a></h3>
<p>For later usage, we can save the trained model.
We only need to specify the name of our model and the file path.
-This call stores the weights and biases of our model.</p>
-<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span>
+This call stores the weights and biases of our model.
+Similarly the transformation metadata to transform input data to the model,
+is saved.</p>
+<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="c1"># Write metadata and trained network to disk.</span>
+ <span class="n">sds</span><span class="o">.</span><span class="n">combine</span><span class="p">(</span>
+ <span class="n">network</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/network'</span><span class="p">),</span>
+ <span class="n">M1</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_X'</span><span class="p">),</span>
+ <span class="n">M2</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_Y'</span><span class="p">)</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">compute</span><span class="p">()</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="step-5-predict-on-unseen-data">
+<h3>Step 5: Predict on Unseen data<a class="headerlink" href="#step-5-predict-on-unseen-data" title="Permalink to this headline">¶</a></h3>
+<p>Once the model is saved along with metadata, it is simple to apply it all to
+unseen data:</p>
+<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="c1"># Read metadata and trained network and do prediction.</span>
+ <span class="n">M1_r</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_X'</span><span class="p">)</span>
+ <span class="n">M2_r</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_Y'</span><span class="p">)</span>
+ <span class="n">network_r</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/network'</span><span class="p">)</span>
+ <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_data</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M1_r</span><span class="p">)</span>
+ <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_labels</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M2_r</span><span class="p">)</span>
+ <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span>
+ <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span>
+ <span class="n">FFN_package_2</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">source</span><span class="p">(</span><span class="n">neural_net_src_path</span><span class="p">,</span> <span class="s2">"fnn"</span><span class="p">)</span>
+ <span class="n">probs</span> <span class="o">=</span> <span class="n">FFN_package_2</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">Xt</span><span class="p">,</span> <span class="n">network_r</span><span class="p">)</span>
+ <span class="n">accuracy</span> <span class="o">=</span> <span class="n">FFN_package_2</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">probs</span><span class="p">,</span> <span class="n">Yt</span><span class="p">)</span><span class="o">.</span><span class="n">compute</span><span class="p">()</span>
</pre></div>
</div>
-<p>Full Script NN
-~~~~~~~~~~~—</p>
+</div>
+<div class="section" id="full-script-nn">
+<h3>Full Script NN<a class="headerlink" href="#full-script-nn" title="Permalink to this headline">¶</a></h3>
<p>The complete script now can be seen here:</p>
<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">systemds.context</span> <span class="kn">import</span> <span class="n">SystemDSContext</span>
<span class="kn">from</span> <span class="nn">systemds.examples.tutorials.adult</span> <span class="kn">import</span> <span class="n">DataManager</span>
@@ -437,15 +464,11 @@ This call stores the weights and biases of our model.</p>
<span class="c1"># Transform frames to matrices.</span>
<span class="n">X</span><span class="p">,</span> <span class="n">M1</span> <span class="o">=</span> <span class="n">X_frame</span><span class="o">.</span><span class="n">transform_encode</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_data</span><span class="p">)</span>
- <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_data</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M1</span><span class="p">)</span>
<span class="n">Y</span><span class="p">,</span> <span class="n">M2</span> <span class="o">=</span> <span class="n">Y_frame</span><span class="o">.</span><span class="n">transform_encode</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_labels</span><span class="p">)</span>
- <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_labels</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M2</span><span class="p">)</span>
<span class="c1"># Subsample to make training faster</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">X</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">train_count</span><span class="p">]</span>
<span class="n">Y</span> <span class="o">=</span> <span class="n">Y</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">train_count</span><span class="p">]</span>
- <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span>
- <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span>
<span class="c1"># Load custom neural network</span>
<span class="n">neural_net_src_path</span> <span class="o">=</span> <span class="s2">"tests/examples/tutorials/neural_net_source.dml"</span>
@@ -458,7 +481,27 @@ This call stores the weights and biases of our model.</p>
<span class="n">network</span> <span class="o">=</span> <span class="n">FFN_package</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Y</span><span class="p">,</span> <span class="n">epochs</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">,</span> <span class="n">learning_rate</span><span class="p">,</span> <span class="n">seed</span><span class="p">)</span>
- <span class="n">network</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/'</span><span class="p">)</span><span class="o">.</span><span class="n">compute</span><span class="p">()</span>
+ <span class="c1"># Write metadata and trained network to disk.</span>
+ <span class="n">sds</span><span class="o">.</span><span class="n">combine</span><span class="p">(</span>
+ <span class="n">network</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/network'</span><span class="p">),</span>
+ <span class="n">M1</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_X'</span><span class="p">),</span>
+ <span class="n">M2</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_Y'</span><span class="p">)</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">compute</span><span class="p">()</span>
+
+ <span class="c1"># Read metadata and trained network and do prediction.</span>
+ <span class="n">M1_r</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_X'</span><span class="p">)</span>
+ <span class="n">M2_r</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_Y'</span><span class="p">)</span>
+ <span class="n">network_r</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/network'</span><span class="p">)</span>
+ <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_data</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M1_r</span><span class="p">)</span>
+ <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_labels</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M2_r</span><span class="p">)</span>
+ <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span>
+ <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span>
+ <span class="n">FFN_package_2</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">source</span><span class="p">(</span><span class="n">neural_net_src_path</span><span class="p">,</span> <span class="s2">"fnn"</span><span class="p">)</span>
+ <span class="n">probs</span> <span class="o">=</span> <span class="n">FFN_package_2</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">Xt</span><span class="p">,</span> <span class="n">network_r</span><span class="p">)</span>
+ <span class="n">accuracy</span> <span class="o">=</span> <span class="n">FFN_package_2</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">probs</span><span class="p">,</span> <span class="n">Yt</span><span class="p">)</span><span class="o">.</span><span class="n">compute</span><span class="p">()</span>
+
+ <span class="kn">import</span> <span class="nn">logging</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"accuracy: "</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">accuracy</span><span class="p">))</span>
</pre></div>
</div>
</div>
diff --git a/docs/api/python/objects.inv b/docs/api/python/objects.inv
index e9899fe28a..402be3e318 100644
Binary files a/docs/api/python/objects.inv and b/docs/api/python/objects.inv differ
diff --git a/docs/api/python/searchindex.js b/docs/api/python/searchindex.js
index 6611d6eb84..6e0d3a2fa6 100644
--- a/docs/api/python/searchindex.js
+++ b/docs/api/python/searchindex.js
@@ -1 +1 @@
-Search.setIndex({docnames:["api/context/systemds_context","api/operator/algorithms","api/operator/node/frame","api/operator/node/list","api/operator/node/matrix","api/operator/node/scalar","api/operator/node/source","api/operator/operation_node","api/script_building/dag","api/script_building/script","api/utils/converters","api/utils/helpers","getting_started/install","getting_started/simple_examples","guide/algorithms_basics","guide/federated","guide/python_end_to_end_tut","index"],envve [...]
\ No newline at end of file
+Search.setIndex({docnames:["api/context/systemds_context","api/operator/algorithms","api/operator/node/frame","api/operator/node/list","api/operator/node/matrix","api/operator/node/scalar","api/operator/node/source","api/operator/operation_node","api/script_building/dag","api/script_building/script","api/utils/converters","api/utils/helpers","getting_started/install","getting_started/simple_examples","guide/algorithms_basics","guide/federated","guide/python_end_to_end_tut","index"],envve [...]
\ No newline at end of file
diff --git a/docs/api/python/sources/guide/python_end_to_end_tut.rst.txt b/docs/api/python/sources/guide/python_end_to_end_tut.rst.txt
index 2a8cb9fb76..961b47d61b 100644
--- a/docs/api/python/sources/guide/python_end_to_end_tut.rst.txt
+++ b/docs/api/python/sources/guide/python_end_to_end_tut.rst.txt
@@ -118,12 +118,13 @@ For this we will introduce another dml file, which can be used to train a basic
Step 1: Obtain data
~~~~~~~~~~~~~~~~~~~
-For the whole data setup please refer to level 1, Step 1, as these steps are identical.
+For the whole data setup please refer to level 1, Step 1, as these steps are almost identical,
+but instead of preparing the test data, we only prepare the training data.
.. include:: ../code/guide/end_to_end/part2.py
:code: python
:start-line: 20
- :end-line: 51
+ :end-line: 47
Step 2: Load the algorithm
~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -134,12 +135,10 @@ This file includes all the necessary functions for training, evaluating, and sto
The returned object of the source call is further used for calling the functions.
The file can be found here:
- - :doc:tests/examples/tutorials/neural_net_source.dml
-
.. include:: ../code/guide/end_to_end/part2.py
:code: python
- :start-line: 54
- :end-line: 55
+ :start-line: 48
+ :end-line: 51
Step 3: Training the neural network
@@ -153,8 +152,8 @@ The seed argument ensures that running the code again yields the same results.
.. include:: ../code/guide/end_to_end/part2.py
:code: python
- :start-line: 61
- :end-line: 62
+ :start-line: 52
+ :end-line: 58
Step 4: Saving the model
@@ -163,15 +162,28 @@ Step 4: Saving the model
For later usage, we can save the trained model.
We only need to specify the name of our model and the file path.
This call stores the weights and biases of our model.
+Similarly the transformation metadata to transform input data to the model,
+is saved.
.. include:: ../code/guide/end_to_end/part2.py
:code: python
- :start-line: 64
+ :start-line: 59
:end-line: 65
+Step 5: Predict on Unseen data
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once the model is saved along with metadata, it is simple to apply it all to
+unseen data:
+
+.. include:: ../code/guide/end_to_end/part2.py
+ :code: python
+ :start-line: 66
+ :end-line: 77
+
Full Script NN
-~~~~~~~~~~~---
+~~~~~~~~~~~~~~
The complete script now can be seen here:
@@ -179,4 +191,4 @@ The complete script now can be seen here:
.. include:: ../code/guide/end_to_end/part2.py
:code: python
:start-line: 20
- :end-line: 64
+ :end-line: 80