You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by IV <ig...@gmail.com> on 2017/02/23 21:54:38 UTC
Ingestion Performance
Hi all,
I'm working on a dashboard which mostly requires OLAP queries and
considering Kylin as analytics engine.
I have the following questions:
1. What latency should I expect while streaming data to Kylin from Kafka?
I.e. how much time could pass between sending message to Kafka and seeing
that this message affects query result?.
2. How much time can it take to build a cube for the following data: 1gb, 5M
entries, 10 dimensions, 5 measures?
3. How much disk space will the cube for the data described in p2 will take?
I understand that all these things depend on setup, hardware and many other
things but I just want to get an idea about Kyling use cases and answers
like sub-second/seconds/tens of seconds/minutes/hours or 5-10x initial data
size are fine for me.
Thanks!
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Ingestion-Performance-tp7287.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: Ingestion Performance
Posted by Li Yang <li...@apache.org>.
> 1. What latency should I expect while streaming data to Kylin from Kafka?
Kylin 1.6 support streaming from Kafka, which uses micro batch to build new
records from Kafka. The speed of micro batch depends on the Kafka volume
and your Hadoop cluster capacity. If done right, the delay could be within
a few minutes, <3 mins for example.
> 2. How much time can it take to build a cube for the following data: 1gb,
5M
entries, 10 dimensions, 5 measures?
Again this depends on your Hadoop cluster. Cube advanced settings (the
aggregation group) matters too. My gut feeling is it should be under 10
minutes for most cases.
> 3. How much disk space will the cube for the data described in p2 will
take?
The expansion rate (cube size / raw data size) could vary from 0.5 to 10,
depending on the cube advanced settings, the compression codec, and your
data set.
On Fri, Feb 24, 2017 at 5:54 AM, IV <ig...@gmail.com> wrote:
> Hi all,
> I'm working on a dashboard which mostly requires OLAP queries and
> considering Kylin as analytics engine.
> I have the following questions:
> 1. What latency should I expect while streaming data to Kylin from Kafka?
> I.e. how much time could pass between sending message to Kafka and seeing
> that this message affects query result?.
> 2. How much time can it take to build a cube for the following data: 1gb,
> 5M
> entries, 10 dimensions, 5 measures?
> 3. How much disk space will the cube for the data described in p2 will
> take?
>
> I understand that all these things depend on setup, hardware and many
> other
> things but I just want to get an idea about Kyling use cases and answers
> like sub-second/seconds/tens of seconds/minutes/hours or 5-10x initial data
> size are fine for me.
>
> Thanks!
>
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Ingestion-Performance-tp7287.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>