You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemml.apache.org by GitBox <gi...@apache.org> on 2020/05/27 12:19:05 UTC
[GitHub] [systemml] Baunsgaard commented on a change in pull request #922: [WIP] Windows developement instructions
Baunsgaard commented on a change in pull request #922:
URL: https://github.com/apache/systemml/pull/922#discussion_r431064703
##########
File path: dev/docs/windows-source-installation.md
##########
@@ -0,0 +1,81 @@
+# Developing Apache SystemDS on Windows Platform
+
+These instructions will help you build Apache SystemDS from source code, which is the basis for engine and algorithms development.
+The following conventions will be used to refer to directories on your machine:
+
+* `<USER_HOME>` is your home directory.
+* `<MAVEN_HOME>` is the root directory for the Apache Maven source code.
+* `<SYSTEMDS_HOME>` is the root directory for the SystemDS source code.
+* `<SPARK_HOME>` is the root directory for the Apache Spark source code.
+* `<SCALA_HOME>` is the root directory for the Scala.
+Ex. `C:\Program Files (x86)\scala`
+* `<SBT_HOME>` is the root directory for the Scala SBT.
Review comment:
Scala And SBT is not needed anymore, since the entire scala part is not in use anymore/ removed for simplicity
##########
File path: dev/docs/windows-source-installation.md
##########
@@ -0,0 +1,81 @@
+# Developing Apache SystemDS on Windows Platform
+
+These instructions will help you build Apache SystemDS from source code, which is the basis for engine and algorithms development.
+The following conventions will be used to refer to directories on your machine:
+
+* `<USER_HOME>` is your home directory.
+* `<MAVEN_HOME>` is the root directory for the Apache Maven source code.
+* `<SYSTEMDS_HOME>` is the root directory for the SystemDS source code.
+* `<SPARK_HOME>` is the root directory for the Apache Spark source code.
+* `<SCALA_HOME>` is the root directory for the Scala.
+Ex. `C:\Program Files (x86)\scala`
+* `<SBT_HOME>` is the root directory for the Scala SBT.
+Ex. `C:\Program Files (x86\sbt`
+* `<JDK_18_HOME>` is the root directory for the 1.8 JDK.
+* `<CUDA_HOME>`,`<CUDA_PATH>` is the top directory for NVIDIA GPU Computing Toolkit.
+Ex. `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0`
+Also, make sure that `libnvvp`, `bin` folders are accessible to command line.
+* `<CUDNN_PATH>` is the path
+Ex. `C:\Program Files\cudnn-9.0-windows10-x64-v7.2.1.38\cuda`
Review comment:
I think this cudnn version is depricated, and we use 10.1 now.
I think a better suggestion is to have a linked folder (if that exists in windows) thereby removing the knowledge about version in the guide
##########
File path: dev/docs/windows-source-installation.md
##########
@@ -0,0 +1,81 @@
+# Developing Apache SystemDS on Windows Platform
+
+These instructions will help you build Apache SystemDS from source code, which is the basis for engine and algorithms development.
+The following conventions will be used to refer to directories on your machine:
+
+* `<USER_HOME>` is your home directory.
+* `<MAVEN_HOME>` is the root directory for the Apache Maven source code.
+* `<SYSTEMDS_HOME>` is the root directory for the SystemDS source code.
+* `<SPARK_HOME>` is the root directory for the Apache Spark source code.
+* `<SCALA_HOME>` is the root directory for the Scala.
+Ex. `C:\Program Files (x86)\scala`
+* `<SBT_HOME>` is the root directory for the Scala SBT.
+Ex. `C:\Program Files (x86\sbt`
+* `<JDK_18_HOME>` is the root directory for the 1.8 JDK.
+* `<CUDA_HOME>`,`<CUDA_PATH>` is the top directory for NVIDIA GPU Computing Toolkit.
+Ex. `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0`
+Also, make sure that `libnvvp`, `bin` folders are accessible to command line.
+* `<CUDNN_PATH>` is the path
+Ex. `C:\Program Files\cudnn-9.0-windows10-x64-v7.2.1.38\cuda`
+
+## Getting Apache SystemDS Source Code
+
+SystemDS source code is available from [github.com/apache/systemml](https://github.com/apache/systemml) by either cloning or
+downloading a zip file (based on a branch) into `<SYSTEMDS_HOME>`. The default is the *master* branch.
+
+````
+git clone https://github.com/apache/systemml systemds
+````
+
+The master branch contains the source code which will be used to create the next major version of Apache SystemDS.
+
+_**Speed Tip:**_ If the complete repository history isn't needed then using a shallow clone (`git clone --depth 1`) will
+save significant time.
+
+[TODO 1]: Get the spark, mvn and java download installation
+Apache SystemDS requires dependencies from the other Apache projects. To download these sources,
+run one of the `getSparkMvnJdk` scripts located in the `<SYSTEMDS_HOME>` directory. These scripts clone their respective
+*master* branches.
+* `getSpark.bat` for Windows.
+
+## Building SystemDS source code
+
+IntelliJ IDEA is preferred for best developer experience.
Review comment:
up for debate :smile: , maybe write it is suggested to use IntelliJ ?
##########
File path: dev/docs/windows-source-installation.md
##########
@@ -0,0 +1,81 @@
+# Developing Apache SystemDS on Windows Platform
+
+These instructions will help you build Apache SystemDS from source code, which is the basis for engine and algorithms development.
+The following conventions will be used to refer to directories on your machine:
+
+* `<USER_HOME>` is your home directory.
+* `<MAVEN_HOME>` is the root directory for the Apache Maven source code.
+* `<SYSTEMDS_HOME>` is the root directory for the SystemDS source code.
+* `<SPARK_HOME>` is the root directory for the Apache Spark source code.
+* `<SCALA_HOME>` is the root directory for the Scala.
+Ex. `C:\Program Files (x86)\scala`
+* `<SBT_HOME>` is the root directory for the Scala SBT.
+Ex. `C:\Program Files (x86\sbt`
+* `<JDK_18_HOME>` is the root directory for the 1.8 JDK.
+* `<CUDA_HOME>`,`<CUDA_PATH>` is the top directory for NVIDIA GPU Computing Toolkit.
+Ex. `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0`
+Also, make sure that `libnvvp`, `bin` folders are accessible to command line.
+* `<CUDNN_PATH>` is the path
+Ex. `C:\Program Files\cudnn-9.0-windows10-x64-v7.2.1.38\cuda`
+
+## Getting Apache SystemDS Source Code
+
+SystemDS source code is available from [github.com/apache/systemml](https://github.com/apache/systemml) by either cloning or
+downloading a zip file (based on a branch) into `<SYSTEMDS_HOME>`. The default is the *master* branch.
+
+````
+git clone https://github.com/apache/systemml systemds
+````
+
+The master branch contains the source code which will be used to create the next major version of Apache SystemDS.
+
+_**Speed Tip:**_ If the complete repository history isn't needed then using a shallow clone (`git clone --depth 1`) will
+save significant time.
+
+[TODO 1]: Get the spark, mvn and java download installation
+Apache SystemDS requires dependencies from the other Apache projects. To download these sources,
+run one of the `getSparkMvnJdk` scripts located in the `<SYSTEMDS_HOME>` directory. These scripts clone their respective
+*master* branches.
+* `getSpark.bat` for Windows.
+
+## Building SystemDS source code
+
+IntelliJ IDEA is preferred for best developer experience.
+
+### Opening the IntelliJ Source Code for Build
+
+Using IntelliJ IDEA **File | Open**, select the `<SYSTEMDS_HOME>` directory.
+* If IntelliJ IDEA displays an error about a missing or out of date required plugin (e.g. maven),
+ [enable, upgrade, or install that plugin](https://www.jetbrains.com/help/idea/managing-plugins.html) and restart IntelliJ IDEA.
+
+### IntelliJ Build Configuration
+
+JDK version 1.8 (u151 or newer) is required for building and developing for SystemDS developement.
+
+1. Using IntelliJ IDEA, [configure](https://www.jetbrains.com/help/idea/sdk.html) a JDK named "**1.8**", pointing to `<JDK_18_HOME>`.
+ * If not already present, add `<JDK_18_HOME>/lib/tools.jar` [to the Classpath](https://www.jetbrains.com/help/idea/sdk.html#manage_sdks) tab
+ for the **1.8** JDK.
+2. If the _Maven Integration_ plugin is disabled, [add the path variable](https://www.jetbrains.com/help/idea/working-with-projects.html#path-variables)
+ "**MAVEN_REPOSITORY**" pointing to `<USER_HOME>/.m2/repository` directory.
+3. _**Speed Tip:**_ If you have enough RAM on your computer,
+ [configure the compiler settings](https://www.jetbrains.com/help/idea/specifying-compilation-settings.html)
+ to enable the "Compile independent modules in parallel" option. Also set the "User-local build process VM options" to `-Xmx2G`.
+ These changes will greatly reduce the compile time.
+4. Now, selecting the IntelliJ IDEA **Build | Build module 'systemds'** option starts the maven build.
+5. _**Speed Tip:**_ If the development machine have enough RAM, [configure the compiler settings](https://www.jetbrains.com/help/idea/specifying-compilation-settings.html)
+ to enable the "Compile independent modules in parallel" option.
Review comment:
Same speed tip is there twice
##########
File path: dev/docs/windows-source-installation.md
##########
@@ -0,0 +1,81 @@
+# Developing Apache SystemDS on Windows Platform
+
+These instructions will help you build Apache SystemDS from source code, which is the basis for engine and algorithms development.
+The following conventions will be used to refer to directories on your machine:
+
+* `<USER_HOME>` is your home directory.
+* `<MAVEN_HOME>` is the root directory for the Apache Maven source code.
+* `<SYSTEMDS_HOME>` is the root directory for the SystemDS source code.
+* `<SPARK_HOME>` is the root directory for the Apache Spark source code.
+* `<SCALA_HOME>` is the root directory for the Scala.
+Ex. `C:\Program Files (x86)\scala`
+* `<SBT_HOME>` is the root directory for the Scala SBT.
+Ex. `C:\Program Files (x86\sbt`
+* `<JDK_18_HOME>` is the root directory for the 1.8 JDK.
+* `<CUDA_HOME>`,`<CUDA_PATH>` is the top directory for NVIDIA GPU Computing Toolkit.
+Ex. `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0`
+Also, make sure that `libnvvp`, `bin` folders are accessible to command line.
+* `<CUDNN_PATH>` is the path
+Ex. `C:\Program Files\cudnn-9.0-windows10-x64-v7.2.1.38\cuda`
+
+## Getting Apache SystemDS Source Code
+
+SystemDS source code is available from [github.com/apache/systemml](https://github.com/apache/systemml) by either cloning or
+downloading a zip file (based on a branch) into `<SYSTEMDS_HOME>`. The default is the *master* branch.
+
+````
+git clone https://github.com/apache/systemml systemds
+````
+
+The master branch contains the source code which will be used to create the next major version of Apache SystemDS.
+
+_**Speed Tip:**_ If the complete repository history isn't needed then using a shallow clone (`git clone --depth 1`) will
+save significant time.
+
+[TODO 1]: Get the spark, mvn and java download installation
+Apache SystemDS requires dependencies from the other Apache projects. To download these sources,
+run one of the `getSparkMvnJdk` scripts located in the `<SYSTEMDS_HOME>` directory. These scripts clone their respective
+*master* branches.
+* `getSpark.bat` for Windows.
+
+## Building SystemDS source code
+
+IntelliJ IDEA is preferred for best developer experience.
+
+### Opening the IntelliJ Source Code for Build
+
+Using IntelliJ IDEA **File | Open**, select the `<SYSTEMDS_HOME>` directory.
+* If IntelliJ IDEA displays an error about a missing or out of date required plugin (e.g. maven),
+ [enable, upgrade, or install that plugin](https://www.jetbrains.com/help/idea/managing-plugins.html) and restart IntelliJ IDEA.
+
+### IntelliJ Build Configuration
+
+JDK version 1.8 (u151 or newer) is required for building and developing for SystemDS developement.
+
+1. Using IntelliJ IDEA, [configure](https://www.jetbrains.com/help/idea/sdk.html) a JDK named "**1.8**", pointing to `<JDK_18_HOME>`.
+ * If not already present, add `<JDK_18_HOME>/lib/tools.jar` [to the Classpath](https://www.jetbrains.com/help/idea/sdk.html#manage_sdks) tab
+ for the **1.8** JDK.
+2. If the _Maven Integration_ plugin is disabled, [add the path variable](https://www.jetbrains.com/help/idea/working-with-projects.html#path-variables)
+ "**MAVEN_REPOSITORY**" pointing to `<USER_HOME>/.m2/repository` directory.
+3. _**Speed Tip:**_ If you have enough RAM on your computer,
+ [configure the compiler settings](https://www.jetbrains.com/help/idea/specifying-compilation-settings.html)
+ to enable the "Compile independent modules in parallel" option. Also set the "User-local build process VM options" to `-Xmx2G`.
+ These changes will greatly reduce the compile time.
+4. Now, selecting the IntelliJ IDEA **Build | Build module 'systemds'** option starts the maven build.
+5. _**Speed Tip:**_ If the development machine have enough RAM, [configure the compiler settings](https://www.jetbrains.com/help/idea/specifying-compilation-settings.html)
+ to enable the "Compile independent modules in parallel" option.
+
+### Building the Source Code
+To build SystemDS from source, choose **Build | Build Project** from the main menu.
+OR
+To maven build, run the `mvn` command in `<SYSTEMDS_HOME>` directory. See the `pom.xml` file for details.
+
+## Testing
+To run the SystemDS built from source, choose **Run | Run** from the main menu.
+
+To run tests on the build, apply these setting to the **Run | Edit Configurations... | Defaults | JUnit** configuration tab:
+ * Working dir: `<SYSTEMDS_HOME>`
+ * VM options:
+ * `-ea`
+ * `-junit4`
+ * `-Djava.system.class.loader=com.intellij.rt.execution.junit.JUnitStarter`
Review comment:
correct me if i'm wrong. But I'm not sure if this executes the tests like the current setup.
It seems to me this is directly using junit for the execution, if that is correct then I'm not sure if it efficiently parallelize the execution of the tests, like if executed with the mvn surefire setup we use in integration testing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org