Update Spark Configuration to improve SystemML performance

[arijit chakraborty <ak...@hotmail.com>]

Hi,


I tried to work on spark configuration file to improve the systemML performance. Even after much tuning R code is running in 40 mins, but SystemML is taking 2.2 hours. Please find the spark configuration screenshots. Please let me know if I'm making some mistake in tuning of the spark configuration. One problem we could rectify is garbage time error. Now, it's completely not there. That was one major bottleneck which was making the code extremely slow.


I'm  working in local system and created a standalone version of spark, with master and workers. The following are the details:


[cid:848caeff-0726-4375-94ea-4874fd1b5f29]



[cid:e03403d6-c902-45a9-8a40-4c2687a1a527]


[cid:4e754a3d-78b1-4462-b8a1-eba7ec403d66]


I also wants to know is it possible to get involved with systemML development? My project is almost on the verge of completion and I learned a lot from you all people. And I really liked this project. So I want to contribute more fruitfully in it.


Thank you!

Arijit