You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@s2graph.apache.org by Min-Seok Kim <ms...@apache.org> on 2016/03/24 03:58:06 UTC

Batch & Speed(Streaming) Layer for S2Graph

[image: pasted1]
(The figure shows S2Graph as lambda architecture view. As mentioned in
[S2GRAPH-15] <https://issues.apache.org/jira/browse/S2GRAPH-15>, S2Graph
provides a great real-time view with serving layer on HBase.)

Hi folks.

I think we'd better implement "the common Spark environment" for batch and
speed(streaming) layer.
This environment is valuable for loader and s2counter which are currently
parts of S2Graph as well as other jobs in the project roadmap(draft:
http://markmail.org/message/uigl6pbt6urelgma?q=list:org%2Eapache%2Es2graph).

In my opinion, the environment includes three important features:
  1. Spark launcher with HA and fault tolerance scheduler using such as
Marathon or Chronos.
  2. Resumable Kafka stream (at least once or exact once)
  3. Configurable Engines like predictionio

Thanks

Minseok

Re: Batch & Speed(Streaming) Layer for S2Graph

Posted by "Kim, Min-Seok" <ms...@gmail.com>.

sorry for the hassle


Minseok

On Mon, Mar 28, 2016 at 5:18 PM, Min-Seok Kim <ms...@apache.org> wrote:

> [image: s2graph-dataflow.png]attaching the missing image
>
>
>
> 2016년 3월 24일 (목) 오전 11:58, Min-Seok Kim <ms...@apache.org>님이 작성:
>
>> [image: pasted1]
>> (The figure shows S2Graph as lambda architecture view. As mentioned in
>> [S2GRAPH-15] <https://issues.apache.org/jira/browse/S2GRAPH-15>, S2Graph
>> provides a great real-time view with serving layer on HBase.)
>>
>> Hi folks.
>>
>> I think we'd better implement "the common Spark environment" for batch
>> and speed(streaming) layer.
>> This environment is valuable for loader and s2counter which are currently
>> parts of S2Graph as well as other jobs in the project roadmap(draft:
>> http://markmail.org/message/uigl6pbt6urelgma?q=list:org%2Eapache%2Es2graph
>> ).
>>
>> In my opinion, the environment includes three important features:
>>   1. Spark launcher with HA and fault tolerance scheduler using such as
>> Marathon or Chronos.
>>   2. Resumable Kafka stream (at least once or exact once)
>>   3. Configurable Engines like predictionio
>>
>> Thanks
>>
>> Minseok
>>
>>
>>
>>

Re: Batch & Speed(Streaming) Layer for S2Graph

Posted by Min-Seok Kim <ms...@apache.org>.

[image: s2graph-dataflow.png]attaching the missing image



2016년 3월 24일 (목) 오전 11:58, Min-Seok Kim <ms...@apache.org>님이 작성:

> [image: pasted1]
> (The figure shows S2Graph as lambda architecture view. As mentioned in
> [S2GRAPH-15] <https://issues.apache.org/jira/browse/S2GRAPH-15>, S2Graph
> provides a great real-time view with serving layer on HBase.)
>
> Hi folks.
>
> I think we'd better implement "the common Spark environment" for batch and
> speed(streaming) layer.
> This environment is valuable for loader and s2counter which are currently
> parts of S2Graph as well as other jobs in the project roadmap(draft:
> http://markmail.org/message/uigl6pbt6urelgma?q=list:org%2Eapache%2Es2graph
> ).
>
> In my opinion, the environment includes three important features:
>   1. Spark launcher with HA and fault tolerance scheduler using such as
> Marathon or Chronos.
>   2. Resumable Kafka stream (at least once or exact once)
>   3. Configurable Engines like predictionio
>
> Thanks
>
> Minseok
>
>
>
>