You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Wang Ken (JIRA)" <ji...@apache.org> on 2016/06/02 15:23:59 UTC

[jira] [Commented] (KYLIN-1726) Scalable streaming cubing

    [ https://issues.apache.org/jira/browse/KYLIN-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312468#comment-15312468 ] 

Wang Ken commented on KYLIN-1726:
---------------------------------

Hi, Yang

Let's meet and discuss how to achieve this. Generally speaking, consuming Kafka from MR is doable. 
But I don't know why you want to add Kafka offset to segment's metadata, Kafka offset is per partition, for each kafka topic, there could be at most hundreds of partitions.
I don't think we should create segment per Kafka partition per Topic.

Thanks
Ken

> Scalable streaming cubing
> -------------------------
>
>                 Key: KYLIN-1726
>                 URL: https://issues.apache.org/jira/browse/KYLIN-1726
>             Project: Kylin
>          Issue Type: New Feature
>            Reporter: liyang
>            Assignee: liyang
>
> We try to achieve:
> 1. Scale streaming cubing workload on a computation cluster, e.g. YARN
> 2. Support Kafka as a formal data source
> 3. Guarantee no data loss reading from Kafka, even records are not strictly ordered by time



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)