You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Wang Ken (JIRA)" <ji...@apache.org> on 2016/06/02 15:23:59 UTC
[jira] [Commented] (KYLIN-1726) Scalable streaming cubing
[ https://issues.apache.org/jira/browse/KYLIN-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312468#comment-15312468 ]
Wang Ken commented on KYLIN-1726:
---------------------------------
Hi, Yang
Let's meet and discuss how to achieve this. Generally speaking, consuming Kafka from MR is doable.
But I don't know why you want to add Kafka offset to segment's metadata, Kafka offset is per partition, for each kafka topic, there could be at most hundreds of partitions.
I don't think we should create segment per Kafka partition per Topic.
Thanks
Ken
> Scalable streaming cubing
> -------------------------
>
> Key: KYLIN-1726
> URL: https://issues.apache.org/jira/browse/KYLIN-1726
> Project: Kylin
> Issue Type: New Feature
> Reporter: liyang
> Assignee: liyang
>
> We try to achieve:
> 1. Scale streaming cubing workload on a computation cluster, e.g. YARN
> 2. Support Kafka as a formal data source
> 3. Guarantee no data loss reading from Kafka, even records are not strictly ordered by time
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)