You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "15927907987@163.com" <15...@163.com> on 2022/02/16 06:59:30 UTC
Which manufacturers' GPUs support Spark?
Hello,
We have done some Spark GPU accelerated work using the spark-rapids component(https://github.com/NVIDIA/spark-rapids). However, we found that this component currently only supports Nvidia GPU, and on the official Spark website, we did not see the manufacturer's description of the GPU supported by spark(https://spark.apache.org/docs/3.2.1/configuration.html#custom-resource-scheduling-and-configuration-overview). So, Can Spark also support GPUs from other manufacturers? such as AMD.
Looking forward to your reply.
15927907987@163.com
Re: Which manufacturers' GPUs support Spark?
Posted by Gourav Sengupta <go...@gmail.com>.
Hi,
100% agree with Sean, the entire RAPIDS solution is built by wonderful
people from NVIDIA.
Just out of curiosity, if you are using AWS then EMR already supports
RAPIDS, please try to use that. AWS has cheaper GPU's which can be used for
testing solutions.
For certain operations on SPARK the GPU's work fantastically well.
Regards,
Gourav Sengupta
On Wed, Feb 16, 2022 at 1:09 PM Sean Owen <sr...@gmail.com> wrote:
> Spark itself does not use GPUs, and is agnostic to what GPUs exist on a
> cluster, scheduled by the resource manager, and used by an application.
> In practice, virtually all GPU-related use cases (for deep learning for
> example) use CUDA, and this is NVIDIA-specific. Certainly, RAPIDS is from
> NVIDIA.
>
> On Wed, Feb 16, 2022 at 7:03 AM 15927907987@163.com <15...@163.com>
> wrote:
>
>> Hello,
>> We have done some Spark GPU accelerated work using the spark-rapids
>> component(https://github.com/NVIDIA/spark-rapids). However, we found
>> that this component currently only supports Nvidia GPU, and on the official
>> Spark website, we did not see the manufacturer's description of the GPU
>> supported by spark(
>> https://spark.apache.org/docs/3.2.1/configuration.html#custom-resource-scheduling-and-configuration-overview).
>> So, Can Spark also support GPUs from other manufacturers? such as AMD.
>> Looking forward to your reply.
>>
>> ------------------------------
>> 15927907987@163.com
>>
>
Re: Which manufacturers' GPUs support Spark?
Posted by Sean Owen <sr...@gmail.com>.
Spark itself does not use GPUs, and is agnostic to what GPUs exist on a
cluster, scheduled by the resource manager, and used by an application.
In practice, virtually all GPU-related use cases (for deep learning for
example) use CUDA, and this is NVIDIA-specific. Certainly, RAPIDS is from
NVIDIA.
On Wed, Feb 16, 2022 at 7:03 AM 15927907987@163.com <15...@163.com>
wrote:
> Hello,
> We have done some Spark GPU accelerated work using the spark-rapids
> component(https://github.com/NVIDIA/spark-rapids). However, we found that
> this component currently only supports Nvidia GPU, and on the official
> Spark website, we did not see the manufacturer's description of the GPU
> supported by spark(
> https://spark.apache.org/docs/3.2.1/configuration.html#custom-resource-scheduling-and-configuration-overview).
> So, Can Spark also support GPUs from other manufacturers? such as AMD.
> Looking forward to your reply.
>
> ------------------------------
> 15927907987@163.com
>
Re: Which manufacturers' GPUs support Spark?
Posted by Sean Owen <sr...@gmail.com>.
Spark itself does not use GPUs, and is agnostic to what GPUs exist on a
cluster, scheduled by the resource manager, and used by an application.
In practice, virtually all GPU-related use cases (for deep learning for
example) use CUDA, and this is NVIDIA-specific. Certainly, RAPIDS is from
NVIDIA.
On Wed, Feb 16, 2022 at 7:03 AM 15927907987@163.com <15...@163.com>
wrote:
> Hello,
> We have done some Spark GPU accelerated work using the spark-rapids
> component(https://github.com/NVIDIA/spark-rapids). However, we found that
> this component currently only supports Nvidia GPU, and on the official
> Spark website, we did not see the manufacturer's description of the GPU
> supported by spark(
> https://spark.apache.org/docs/3.2.1/configuration.html#custom-resource-scheduling-and-configuration-overview).
> So, Can Spark also support GPUs from other manufacturers? such as AMD.
> Looking forward to your reply.
>
> ------------------------------
> 15927907987@163.com
>