You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Boric Tan <it...@gmail.com> on 2016/01/20 01:12:46 UTC
How Spark utilize low-level architecture features?
Hi there,
I am new to Spark, and would like to get some help to understand if Spark
can utilize the underlying architectures for better performance. If so, how
does it do it?
For example, assume there is a cluster built with machines of different
CPUs, will Spark check the individual CPU information and use some
machine-specific setting for the tasks assigned to that machine? Or is it
totally dependent on the underlying JVM implementation to run the JAR file,
and therefor the JVM is the place to check if certain CPU features can be
used?
Thanks,
Boric
Re: How Spark utilize low-level architecture features?
Posted by borictan <it...@gmail.com>.
Thanks, Fokko.
Yes, increasing the parallelism is one way to speed up the performance. On
the other hand, we are also looking for opportunities to harness the single
node hardware power to increase the single-node performance, which will help
the overall performance.
Thanks,
Boric
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/How-Spark-utilize-low-level-architecture-features-tp16052p16073.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: How Spark utilize low-level architecture features?
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi Boric,
For the Spark Mllib package, which is build on top of Breeze
<https://github.com/scalanlp/breeze>, which uses in turn netlib-java
<https://github.com/fommil/netlib-java>. This netlib-java library can be
optimized for each system by compiling the specific architecture:
*To get optimal performance for a specific machine, it is best to compile
locally by grabbing the latest ATLAS or the latest OpenBLAS and following
the compilation instructions.*
For the rest, Spark focusses on adding more machines instead of using very
specific optimization procedures. Also optimizing your jobs (decreasing
communication between workers e.d.) might do the trick.
Cheers, Fokko.
2016-01-21 6:55 GMT+01:00 Boric Tan <it...@gmail.com>:
> Anyone could shed some light on this?
>
> Thanks,
> Boric
>
> On Tue, Jan 19, 2016 at 4:12 PM, Boric Tan <it...@gmail.com>
> wrote:
>
>> Hi there,
>>
>> I am new to Spark, and would like to get some help to understand if Spark
>> can utilize the underlying architectures for better performance. If so, how
>> does it do it?
>>
>> For example, assume there is a cluster built with machines of different
>> CPUs, will Spark check the individual CPU information and use some
>> machine-specific setting for the tasks assigned to that machine? Or is it
>> totally dependent on the underlying JVM implementation to run the JAR file,
>> and therefor the JVM is the place to check if certain CPU features can be
>> used?
>>
>> Thanks,
>> Boric
>>
>
>
Re: How Spark utilize low-level architecture features?
Posted by Boric Tan <it...@gmail.com>.
Anyone could shed some light on this?
Thanks,
Boric
On Tue, Jan 19, 2016 at 4:12 PM, Boric Tan <it...@gmail.com> wrote:
> Hi there,
>
> I am new to Spark, and would like to get some help to understand if Spark
> can utilize the underlying architectures for better performance. If so, how
> does it do it?
>
> For example, assume there is a cluster built with machines of different
> CPUs, will Spark check the individual CPU information and use some
> machine-specific setting for the tasks assigned to that machine? Or is it
> totally dependent on the underlying JVM implementation to run the JAR file,
> and therefor the JVM is the place to check if certain CPU features can be
> used?
>
> Thanks,
> Boric
>
Re: How Spark utilize low-level architecture features?
Posted by borictan <it...@gmail.com>.
Thanks for the explanation, Steve.
I don't want to control where the work is done. What I wanted to understand
is if Spark could take advantage of the underlying architecture features.
For example, if the CPUs on the nodes support some improved vector
instructions, can the Spark jobs (if they have a lot of vector operations)
benefit from this? If yes, how does it happen, inside Spark, or the JVM
where the the job TAR is running on?
Also, for the GPU part you mentioned, labeling the GPU nodes, and scheduling
work to those GPU-enabled system does not mean the GPU computation power
will be utilized, right? The user has to provide CUDE codes
(openCL/CUDA/etc) and somehow link them to the system. Is my understanding
correct?
Thanks,
Boric
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/How-Spark-utilize-low-level-architecture-features-tp16052p16072.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: How Spark utilize low-level architecture features?
Posted by Steve Loughran <st...@hortonworks.com>.
> On 19 Jan 2016, at 16:12, Boric Tan <it...@gmail.com> wrote:
>
> Hi there,
>
> I am new to Spark, and would like to get some help to understand if Spark can utilize the underlying architectures for better performance. If so, how does it do it?
>
> For example, assume there is a cluster built with machines of different CPUs, will Spark check the individual CPU information and use some machine-specific setting for the tasks assigned to that machine? Or is it totally dependent on the underlying JVM implementation to run the JAR file, and therefor the JVM is the place to check if certain CPU features can be used?
>
> Thanks,
> Boric
You can't control where work is done based on CPU parts. Ideally your cluster is homogenous, or at least vary only in CPU performance and memory.
If some of your systems have GPUs and some don't, then in a YARN cluster, label the GPU parts and then use yarn queues or spark-submit to schedule the work only on those GPU systems.
The native libraries you load into JVMs are generally where CPU checks and features (e.g. x86 AES opcodes for encrypt/decrypt) would go
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org