[jira] [Commented] (SYSTEMML-1451) Automate performance testing and reporting

["Krishna Kalyan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SYSTEMML-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998541#comment-15998541 ] 

Krishna Kalyan commented on SYSTEMML-1451:
------------------------------------------

Hello [~nakul02], 
This is a performance test for `genBinomialData` with less than `80` MB data. Yes, the default config is set to high for my PC. I tried using this to  limit jvm usage `export JAVA_OPTS="-Xmx2G -Xms2g -Xmn1g"`. However my script still fails. Could you please guide on how to set limits via the environment variables?/ configuration file?.

Regards,
Krishna

> Automate performance testing and reporting
> ------------------------------------------
>
>                 Key: SYSTEMML-1451
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1451
>             Project: SystemML
>          Issue Type: Improvement
>          Components: Infrastructure, Test
>            Reporter: Nakul Jindal
>              Labels: gsoc2017, mentor, performance, reporting, testing
>
> As part of a release (and in general), performance tests are run for SystemML.
> Currently, running and reporting on these performance tests are a manual process. There are helper scripts, but largely the process is manual.
> The aim of this GSoC 2017 project is to automate performance testing and its reporting.
> These are the tasks that this entails
> 1. Automate running of the performance tests, including generation of test data
> 2. Detect errors and report if any
> 3. Record performance benchmarking information
> 4. Automatically compare this performance to previous version to check for performance regressions
> 5. Automatically compare to Spark MLLib, R?, Julia?
> 6. Prepare report with all the information about failed jobs, performance information, perf info against other comparable projects/algorithms (plotted/in plain text in CSV, PDF or other common format)
> 7. Create scripts to automatically run this process on a cloud provider that spins up machines, runs the test, saves the reports and spins down the machines.
> 8. Create a web application to do this interactively without dropping down into a shell.
> As part of this project, the student will need to know scripting (in Bash, Python, etc). It may also involve changing error reporting and performance reporting code in SystemML. 
> Rating - Medium (for the amount of work)
> Mentor - [~nakul02] (Other co-mentors will join in)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)