You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by qingyang li <li...@gmail.com> on 2014/05/15 10:15:44 UTC
can RDD be shared across mutil spark applications?
Re: can RDD be shared across mutil spark applications?
Posted by qingyang li <li...@gmail.com>.
thanks for sharing, I am using tachyon to store RDD now.
2014-05-18 12:02 GMT+08:00 Christopher Nguyen <ct...@adatao.com>:
> Qing Yang, Andy is correct in answering your direct question.
>
> At the same time, depending on your context, you may be able to apply a
> pattern where you turn the single Spark application into a service, and
> multiple clients if that service can indeed share access to the same RDDs.
>
> Several groups have built apps based on this pattern, and we will also show
> something with this behavior at the upcoming Spark Summit (multiple users
> collaborating on named DDFs with the same underlying RDDs).
>
> Sent while mobile. Pls excuse typos etc.
> On May 18, 2014 9:40 AM, "Andy Konwinski" <an...@gmail.com> wrote:
>
> > RDDs cannot currently be shared across multiple SparkContexts without
> using
> > something like the Tachyon project (which is a separate
> project/codebase).
> >
> > Andy
> > On May 16, 2014 2:14 PM, "qingyang li" <li...@gmail.com> wrote:
> >
> > >
> > >
> >
>
Re: can RDD be shared across mutil spark applications?
Posted by Christopher Nguyen <ct...@adatao.com>.
Qing Yang, Andy is correct in answering your direct question.
At the same time, depending on your context, you may be able to apply a
pattern where you turn the single Spark application into a service, and
multiple clients if that service can indeed share access to the same RDDs.
Several groups have built apps based on this pattern, and we will also show
something with this behavior at the upcoming Spark Summit (multiple users
collaborating on named DDFs with the same underlying RDDs).
Sent while mobile. Pls excuse typos etc.
On May 18, 2014 9:40 AM, "Andy Konwinski" <an...@gmail.com> wrote:
> RDDs cannot currently be shared across multiple SparkContexts without using
> something like the Tachyon project (which is a separate project/codebase).
>
> Andy
> On May 16, 2014 2:14 PM, "qingyang li" <li...@gmail.com> wrote:
>
> >
> >
>
Re: can RDD be shared across mutil spark applications?
Posted by Andy Konwinski <an...@gmail.com>.
RDDs cannot currently be shared across multiple SparkContexts without using
something like the Tachyon project (which is a separate project/codebase).
Andy
On May 16, 2014 2:14 PM, "qingyang li" <li...@gmail.com> wrote:
>
>