You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lens.apache.org by Santoshakhilesh <sa...@huawei.com> on 2016/07/08 03:03:04 UTC
Few Questions about Lens
Hi Lens Experts ,
I have come through this project while looking for an OLAP solution on Hadoop Stack for my product. I have few queries , please someone answer them.
To give background of my requirement.
1. My system is collecting performance KPIs which we collect and calculate and want to store in HBase.
2. Frequency and size of my data is every 5 minutes around 5 million rows and each row about 30 - 40 KPIs (this is current load and can increase in future)
3. I need to support following type of queries
a) Querying raw KPI data based on filters and joins and order by clauses - Expected latency < 2 seconds
b) OLAP queries with groupby and aggregators like sum , max , avg etc. - Expected latency < 2 seconds
c) TOP N , Bottom N
d) For large scans paging
4. Large scans based on filters for Machine Learning and analytics Job
Now my questions are ; (I have read the Lens documents but I still have some confusions)
1. Lens is ROLAP solution ? - Which is the primary source of meta data and real data
2. Lens is MOLAP solution ? - How do I build cubes ? whats the latency of building cubes (any benchmarking available)
3. Which of my requirements can be met by Lens ?
4. Are there any commercial deployment with Lens ?
Thanks in advance.
Regards,
Santosh Akhilesh
Re: Few Questions about Lens
Posted by Puneet Gupta <pu...@inmobi.com>.
Hi Santosh
Sorry , I could not respond earlier.
Lens supports OLAP but does to restrict the user to just OLAP. So you can
fire queries even without creating a cube. In this case server will check
each driver(incase there are many, say one for MySql, One for Hive) on
ability to execute the issued query and then pick the correct driver. But
in a production system having a cube will be a better option as it not
only provides users with easier querying but also provides lots of features
that are supported via cube (like calculated expressions, start time for
columns, display names, data availability, correct fact resolution,etc)
You can download the binary here
<http://lens.apache.org/releases/download.html> (lens version 2.5)
You can follow the steps for installation and running examples here :
http://lens.apache.org/lenshome/install-and-run.html
There is a demo video here
<https://cwiki.apache.org/confluence/display/LENS/2015/07/13/20+Minute+video+demo+of+Apache+Lens+through+examples>
Note: Lens has Hadoop and Hive as dependencies.
If you have an existing Hadoop deployment (2.6 and above), you can use it
directly. Just set HADOOP_HOME .
For Hive we depend on a custom version
<https://github.com/InMobi/hive/releases/tag/hive-release-0.13.4-inm> of
Hive which has some extra optimizations compared to apache Hive. Lens
version 2.6 will be out around August mid, where we have committed all
hive optimizations to apache hive 2.x.x and Lens 2.6 will directly depend
on Apache Hive 2.x.x in that case.
Thanks,
Puneet Gupta
On Mon, Jul 11, 2016 at 11:52 AM, Santoshakhilesh <
santosh.akhilesh@huawei.com> wrote:
> HI Puneet ,
>
> Thanks for answering the queries.
>
> Lens being OLAP , does it mean I actually don’t need to build cubes , the
> data can reside on sources you mentioned and Lens can be used as ROLAP
> engine for query ?
>
> Can you point out to a link where I can download and test sample data
> sources and use cases.
>
> Regards,
>
> Santosh Akhilesh
>
>
>
>
>
> *From:* Puneet Gupta [mailto:puneet.gupta@inmobi.com]
> *Sent:* 08 July 2016 15:25
> *To:* Santoshakhilesh
> *Cc:* user@lens.apache.org
> *Subject:* Re: Few Questions about Lens
>
>
>
> Hi Santosh
>
>
>
> I would try to answer your questions .
>
>
>
> To begin with, lens has two large scale deployments (which crunch PBs
> level data) , one at InMobi and another at Flipkart that I know of. Apart
> from this there are some smaller deployments too.
>
>
>
>
>
> The main usp of lens is
>
>
>
> 1. Seamless access to data that sits in multiple data sources (hive,
> spark, mysql, vertica, druid, elastic search, etc - more can be plugged in)
> in a lamda architected analytics system. Lens can make the right choice of
> data for any user query based on the data availability in underlying data
> storages and their response time.
>
>
>
> 2. Provide an olap interface to query the data. This hides all the
> complexity from end user who can write simple queries over cubes and
> underneath lens will translate the queries by joining the right facts with
> dimensions and resolving the fields to complex expressions. The olap
> interface is very flexible and allows the data schema to change with time.
> The schema allows for horizontal or vertical split/distribution of fact
> tables (vertical split is in works and will be out soon)
>
> So i would say Lens is an ROLAP system but can have data stored in
> multiple data sources compared to a traditional ROLAP that supports simple
> storage.
>
>
>
> 3.Provide quota management for users and thus regulate the usage of lens
> server to offer fair share to each user.
>
>
>
> 4. Provides multiple ways to communicate with lens sever via REST API,
> Java client, python client and a lens-cli. A slack bot is also active, but
> it not yet open sourced.
>
>
>
>
>
> For your use case involving Hbase, see if you can try and use the lens
> JDBC driver with phoenix <https://phoenix.apache.org/> . Your data
> ingestion and querying capacity will depend on HBase and phoenix in this
> case. You can also chose to implement a lens HBase driver if you need
> better control.
>
>
>
>
> Thanks,
>
> Puneet Gupta
>
>
>
> On Fri, Jul 8, 2016 at 8:33 AM, Santoshakhilesh <
> santosh.akhilesh@huawei.com> wrote:
>
> Hi Lens Experts ,
>
>
>
> I have come through this project while looking for an OLAP solution
> on Hadoop Stack for my product. I have few queries , please someone answer
> them.
>
>
>
> To give background of my requirement.
>
>
>
> 1. My system is collecting performance KPIs which we collect and
> calculate and want to store in HBase.
>
> 2. Frequency and size of my data is every 5 minutes around 5
> million rows and each row about 30 – 40 KPIs (this is current load and can
> increase in future)
>
> 3. I need to support following type of queries
>
>
>
> a) Querying raw KPI data based on filters and joins and order by
> clauses – Expected latency < 2 seconds
>
> b) OLAP queries with groupby and aggregators like sum , max , avg
> etc. – Expected latency < 2 seconds
>
> c) TOP N , Bottom N
>
> d) For large scans paging
>
> 4. Large scans based on filters for Machine Learning and analytics
> Job
>
>
>
> Now my questions are ; (I have read the Lens documents but I still have
> some confusions)
>
>
>
> 1. Lens is ROLAP solution ? – Which is the primary source of meta
> data and real data
>
> 2. Lens is MOLAP solution ? – How do I build cubes ? whats the
> latency of building cubes (any benchmarking available)
>
> 3. Which of my requirements can be met by Lens ?
>
> 4. Are there any commercial deployment with Lens ?
>
>
>
> Thanks in advance.
>
>
>
> Regards,
>
> Santosh Akhilesh
>
>
>
>
>
>
>
>
>
>
>
> _____________________________________________________________
>
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Few Questions about Lens
Posted by Puneet Gupta <pu...@inmobi.com>.
Hi Santosh
I would try to answer your questions .
To begin with, lens has two large scale deployments (which crunch PBs level
data) , one at InMobi and another at Flipkart that I know of. Apart from
this there are some smaller deployments too.
The main usp of lens is
1. Seamless access to data that sits in multiple data sources (hive, spark,
mysql, vertica, druid, elastic search, etc - more can be plugged in) in a
lamda architected analytics system. Lens can make the right choice of data
for any user query based on the data availability in underlying data
storages and their response time.
2. Provide an olap interface to query the data. This hides all the
complexity from end user who can write simple queries over cubes and
underneath lens will translate the queries by joining the right facts with
dimensions and resolving the fields to complex expressions. The olap
interface is very flexible and allows the data schema to change with time.
The schema allows for horizontal or vertical split/distribution of fact
tables (vertical split is in works and will be out soon)
So i would say Lens is an ROLAP system but can have data stored in multiple
data sources compared to a traditional ROLAP that supports simple storage.
3.Provide quota management for users and thus regulate the usage of lens
server to offer fair share to each user.
4. Provides multiple ways to communicate with lens sever via REST API, Java
client, python client and a lens-cli. A slack bot is also active, but it
not yet open sourced.
For your use case involving Hbase, see if you can try and use the lens JDBC
driver with phoenix <https://phoenix.apache.org/> . Your data ingestion and
querying capacity will depend on HBase and phoenix in this case. You can
also chose to implement a lens HBase driver if you need better control.
Thanks,
Puneet Gupta
On Fri, Jul 8, 2016 at 8:33 AM, Santoshakhilesh <santosh.akhilesh@huawei.com
> wrote:
> Hi Lens Experts ,
>
>
>
> I have come through this project while looking for an OLAP solution
> on Hadoop Stack for my product. I have few queries , please someone answer
> them.
>
>
>
> To give background of my requirement.
>
>
>
> 1. My system is collecting performance KPIs which we collect and
> calculate and want to store in HBase.
>
> 2. Frequency and size of my data is every 5 minutes around 5
> million rows and each row about 30 – 40 KPIs (this is current load and can
> increase in future)
>
> 3. I need to support following type of queries
>
>
>
> a) Querying raw KPI data based on filters and joins and order by
> clauses – Expected latency < 2 seconds
>
> b) OLAP queries with groupby and aggregators like sum , max , avg
> etc. – Expected latency < 2 seconds
>
> c) TOP N , Bottom N
>
> d) For large scans paging
>
> 4. Large scans based on filters for Machine Learning and analytics
> Job
>
>
>
> Now my questions are ; (I have read the Lens documents but I still have
> some confusions)
>
>
>
> 1. Lens is ROLAP solution ? – Which is the primary source of meta
> data and real data
>
> 2. Lens is MOLAP solution ? – How do I build cubes ? whats the
> latency of building cubes (any benchmarking available)
>
> 3. Which of my requirements can be met by Lens ?
>
> 4. Are there any commercial deployment with Lens ?
>
>
>
> Thanks in advance.
>
>
>
> Regards,
>
> Santosh Akhilesh
>
>
>
>
>
>
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.