You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Andrew Melo <an...@gmail.com> on 2021/05/18 20:36:33 UTC

Secrets store for DSv2

Hello,

When implementing a DSv2 datasource, where is an appropriate place to
store/transmit secrets from the driver to the executors? Is there
built-in spark functionality for that, or is my best bet to stash it
as a member variable in one of the classes that gets sent to the
executors?

Thanks!
Andrew

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: Secrets store for DSv2

Posted by Wenchen Fan <cl...@gmail.com>.

You can take a look at PartitionReaderFactory.

It's created at the driver side, serialized and sent to the executor side.
For the write side, there is a similar channel: DataWriterFactory

On Wed, May 19, 2021 at 4:37 AM Andrew Melo <an...@gmail.com> wrote:

> Hello,
>
> When implementing a DSv2 datasource, where is an appropriate place to
> store/transmit secrets from the driver to the executors? Is there
> built-in spark functionality for that, or is my best bet to stash it
> as a member variable in one of the classes that gets sent to the
> executors?
>
> Thanks!
> Andrew
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>