Question regarding instability of EdgeProperty DataSourceType

[Gal Vinograd <ga...@crosswise.com>]

Hey,

I read though tez-api code and noticed that PERSISTED is the only stable
data source type. Does that mean that data isn't transmitted between
vertices though in-memory or network channels?

Thanks :)

Re: Question regarding instability of EdgeProperty DataSourceType

[Gal Vinograd <ga...@crosswise.com>]

Thank you so much, this is a really well described answer :)

On Sun, Jan 31, 2016 at 7:57 PM Hitesh Shah <hi...@apache.org> wrote:

> There are 3 types defined as you have noticed:
>
> persisted_reliable: assumes a vertex output is stored in a reliable store
> like HDFS. This states that if the node on which the task ran disappears,
> the output is still available.
> persisted: vertex output stored on local disk where the task ran.
> ephemeral: vertex output stored in task memory.
>
> From a data transmission point of view, all data is always transmitted
> over network unless there is a case where the downstream task is running on
> same machine as the task that generated the output. In that case, it can
> read from local disk if needed.
>
> You are right that the in-memory support is not built out so a co-located
> task potentially reading from another task’s memory is therefore not
> supported today. The network channel requires a bit more explanation.
> Generally, all data is persisted to disk. This means that data transferred
> over the network is first written locally and then eventually pulled/pushed
> to a downstream task as needed. This does not mean that all the data needs
> to be generated first before being sent downstream. Data can be still
> generated in “chunks” and then sent downstream as when as a chunk becomes
> available. ( this functionality is internally called “pipelined shuffle” if
> you end up searching through the code/jiras ). However, again to clarify,
> there is no pure streaming support yet where data is kept in memory and
> pushed downstream.
>
> To add, the in-memory approach requires changing Tez to have a different
> fault tolerance model to be applied and it also needs more cluster capacity
> to ensure that both upstream and downstream tasks can run concurrently. Do
> you see this as a requirement for something that you are planning to use
> Tez for?
>
> thanks
> — Hitesh
>
> On Jan 31, 2016, at 12:30 AM, Gal Vinograd <ga...@crosswise.com> wrote:
>
> > Hey,
> >
> > I read though tez-api code and noticed that PERSISTED is the only stable
> data source type. Does that mean that data isn't transmitted between
> vertices though in-memory or network channels?
> >
> > Thanks :)
>
>

Re: Question regarding instability of EdgeProperty DataSourceType

[Hitesh Shah <hi...@apache.org>]

There are 3 types defined as you have noticed:

persisted_reliable: assumes a vertex output is stored in a reliable store like HDFS. This states that if the node on which the task ran disappears, the output is still available. 
persisted: vertex output stored on local disk where the task ran. 
ephemeral: vertex output stored in task memory. 

From a data transmission point of view, all data is always transmitted over network unless there is a case where the downstream task is running on same machine as the task that generated the output. In that case, it can read from local disk if needed.

You are right that the in-memory support is not built out so a co-located task potentially reading from another task’s memory is therefore not supported today. The network channel requires a bit more explanation. Generally, all data is persisted to disk. This means that data transferred over the network is first written locally and then eventually pulled/pushed to a downstream task as needed. This does not mean that all the data needs to be generated first before being sent downstream. Data can be still generated in “chunks” and then sent downstream as when as a chunk becomes available. ( this functionality is internally called “pipelined shuffle” if you end up searching through the code/jiras ). However, again to clarify, there is no pure streaming support yet where data is kept in memory and pushed downstream. 

To add, the in-memory approach requires changing Tez to have a different fault tolerance model to be applied and it also needs more cluster capacity to ensure that both upstream and downstream tasks can run concurrently. Do you see this as a requirement for something that you are planning to use Tez for? 

thanks
— Hitesh

On Jan 31, 2016, at 12:30 AM, Gal Vinograd <ga...@crosswise.com> wrote:

> Hey,
> 
> I read though tez-api code and noticed that PERSISTED is the only stable data source type. Does that mean that data isn't transmitted between vertices though in-memory or network channels?
> 
> Thanks :)