Newbie questions

[Kenneth Tran <kt...@microsoft.com>]

Hi,



1.       Is Spark suited for online learning algorithms? From what I've read so far (mainly from this slide<http://www.slideshare.net/Hadoop_Summit/spark-and-shark>), it seems not but I could be wrong.



2.       This is a Scala/JVM question. How easy is it to interop with native code (C++)? For me, it's important to be able to use MKL, CUDA, and write custom C++ code to utilize SIMD instructions.

(It's hard to talk about distributed computing when we haven't optimized at the single machine level.)



I'm new to Spark so apologize if you find my questions too naïve :).



Thanks.

-Ken

Re: Newbie questions

[Matei Zaharia <ma...@gmail.com>]

Hi Kenneth,

> 1.       Is Spark suited for online learning algorithms? From what I’ve read so far (mainly from this slide), it seems not but I could be wrong.

You can probably use Spark Streaming (http://spark.incubator.apache.org/docs/latest/streaming-programming-guide.html) to implement online algorithms. I know at least one group that implemented an online version of K-means this way. Spark also comes with a machine learning library that currently only has static versions (not for streaming data).

> 2.       This is a Scala/JVM question. How easy is it to interop with native code (C++)? For me, it’s important to be able to use MKL, CUDA, and write custom C++ code to utilize SIMD instructions.
> (It’s hard to talk about distributed computing when we haven’t optimized at the single machine level.)

Java provides widely used facilities to talk to native code through the Java Native Interface (JNI), and wrappers around some common libraries already exist. For example, JBLAS (http://mikiobraun.github.io/jblas/) is a wrapper around BLAS, JavaCL (https://code.google.com/p/javacl/) covers OpenCL, and Intel has some examples on MKL: http://software.intel.com/sites/products/documentation/hpc/mkl/mkl_userguide_lnx/GUID-15EA8C86-7F31-4209-AD45-0D4E903F5445.htm. We use JBLAS in Spark.

Matei