You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Mark Zhao <gu...@gmail.com> on 2019/09/17 14:52:59 UTC

Can I set the Alluxio WriteType in Spark applications?

Hi,

If Spark applications write data into alluxio, can WriteType be configured?

Thanks,
Mark

Re: Can I set the Alluxio WriteType in Spark applications?

Posted by Bin Fan <fa...@gmail.com>.

Hi Mark,

You can follow the instructions here:
https://docs.alluxio.io/os/user/stable/en/compute/Spark.html#customize-alluxio-user-properties-for-individual-spark-jobs

Something like this:

$ spark-submit \--conf
'spark.driver.extraJavaOptions=-Dalluxio.user.file.writetype.default=CACHE_THROUGH'
\--conf 'spark.executor.extraJavaOptions=-Dalluxio.user.file.writetype.default=CACHE_THROUGH'
\...

Hope it helps

- Bin

On Tue, Sep 17, 2019 at 7:53 AM Mark Zhao <gu...@gmail.com> wrote:

> Hi,
>
> If Spark applications write data into alluxio, can WriteType be configured?
>
> Thanks,
> Mark
>
>