You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lens.apache.org by Tao Yan <ty...@linkedin.com> on 2016/08/30 00:48:03 UTC
Lens Scalibility
Hi Lens Developers,
As the number of users grow, one Lens server may not be able to support
all users. Is it possible to setup a multi-server Lens cluster?
Thanks,
--
*Tao Yan*
Software Engineer
Data Analytics Infrastructure Tools and Services
206.250.5345
tyan@linkedin.com
https://www.linkedin.com/in/taousc
Re: Lens Scalibility
Posted by Tao Yan <ty...@linkedin.com>.
Thanks!
On Thu, Sep 1, 2016 at 12:54 AM, Puneet Gupta <pu...@inmobi.com>
wrote:
> As of now lens is self serve and we expect users /secondary developers to
> onboard their schemas on to lens service. So we spend minimal effort in
> supporting users as such.
> There are some internal tools that are made available to our falcon
> <https://falcon.apache.org/> run pipelines to interact with lens service
> for partition registration.
> We use the metrics emitted by lens service (http://lensserver:8080/admin/
> metrics/?pretty=true) to monitor the service health.
>
> Most developers are involved in building new features/improvements/bug
> fixes.
>
>
>
> Thanks,
> Puneet Gupta
>
> On Tue, Aug 30, 2016 at 10:07 PM, Tao Yan <ty...@linkedin.com> wrote:
>
>> Hi Puneet,
>>
>> Thanks very much for the information. Do you mind sharing some
>> information about the number of engineers developing / maintaining Lens?
>> the number of users and the percentage time developers used to support them?
>>
>> Thanks,
>>
>> On Tue, Aug 30, 2016 at 1:18 AM, Puneet Gupta <pu...@inmobi.com>
>> wrote:
>>
>>> Hi Tao
>>>
>>> You can find some discussion on this aspect in LENS-658
>>> <https://issues.apache.org/jira/browse/LENS-658>.
>>>
>>> As of now Lens server does not support clustered setup. From what we
>>> have seen in production, lens server consumes very little system resources
>>> as all the heavy lifting is done by target systems (Hive, Mysql, Vertica).
>>> The only operation on lens server that can consume some resources is query
>>> result formatting. We are able to easily support ~750 unique users and ~100
>>> concurrent users. We are running about 8000 queries per week.
>>>
>>> What is the kind or load your are expecting for your deployment ?
>>>
>>> Thanks,
>>> Puneet Gupta
>>>
>>> On Tue, Aug 30, 2016 at 6:18 AM, Tao Yan <ty...@linkedin.com> wrote:
>>>
>>>> Hi Lens Developers,
>>>>
>>>> As the number of users grow, one Lens server may not be able to support
>>>> all users. Is it possible to setup a multi-server Lens cluster?
>>>>
>>>> Thanks,
>>>> --
>>>>
>>>> *Tao Yan*
>>>> Software Engineer
>>>> Data Analytics Infrastructure Tools and Services
>>>>
>>>>
>>>>
>>>> 206.250.5345
>>>> tyan@linkedin.com
>>>> https://www.linkedin.com/in/taousc
>>>>
>>>
>>>
>>> _____________________________________________________________
>>> The information contained in this communication is intended solely for
>>> the use of the individual or entity to whom it is addressed and others
>>> authorized to receive it. It may contain confidential or legally privileged
>>> information. If you are not the intended recipient you are hereby notified
>>> that any disclosure, copying, distribution or taking any action in reliance
>>> on the contents of this information is strictly prohibited and may be
>>> unlawful. If you have received this communication in error, please notify
>>> us immediately by responding to this email and then delete it from your
>>> system. The firm is neither liable for the proper and complete transmission
>>> of the information contained in this communication nor for any delay in its
>>> receipt.
>>
>>
>>
>>
>> --
>>
>> *Tao Yan*
>> Software Engineer
>> Data Analytics Infrastructure Tools and Services
>>
>>
>>
>> 206.250.5345
>> tyan@linkedin.com
>> https://www.linkedin.com/in/taousc
>>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
--
*Tao Yan*
Software Engineer
Data Analytics Infrastructure Tools and Services
206.250.5345
tyan@linkedin.com
https://www.linkedin.com/in/taousc
Re: Lens Scalibility
Posted by Puneet Gupta <pu...@inmobi.com>.
As of now lens is self serve and we expect users /secondary developers to
onboard their schemas on to lens service. So we spend minimal effort in
supporting users as such.
There are some internal tools that are made available to our falcon
<https://falcon.apache.org/> run pipelines to interact with lens service
for partition registration.
We use the metrics emitted by lens service (
http://lensserver:8080/admin/metrics/?pretty=true) to monitor the service
health.
Most developers are involved in building new features/improvements/bug
fixes.
Thanks,
Puneet Gupta
On Tue, Aug 30, 2016 at 10:07 PM, Tao Yan <ty...@linkedin.com> wrote:
> Hi Puneet,
>
> Thanks very much for the information. Do you mind sharing some information
> about the number of engineers developing / maintaining Lens? the number of
> users and the percentage time developers used to support them?
>
> Thanks,
>
> On Tue, Aug 30, 2016 at 1:18 AM, Puneet Gupta <pu...@inmobi.com>
> wrote:
>
>> Hi Tao
>>
>> You can find some discussion on this aspect in LENS-658
>> <https://issues.apache.org/jira/browse/LENS-658>.
>>
>> As of now Lens server does not support clustered setup. From what we have
>> seen in production, lens server consumes very little system resources as
>> all the heavy lifting is done by target systems (Hive, Mysql, Vertica). The
>> only operation on lens server that can consume some resources is query
>> result formatting. We are able to easily support ~750 unique users and ~100
>> concurrent users. We are running about 8000 queries per week.
>>
>> What is the kind or load your are expecting for your deployment ?
>>
>> Thanks,
>> Puneet Gupta
>>
>> On Tue, Aug 30, 2016 at 6:18 AM, Tao Yan <ty...@linkedin.com> wrote:
>>
>>> Hi Lens Developers,
>>>
>>> As the number of users grow, one Lens server may not be able to support
>>> all users. Is it possible to setup a multi-server Lens cluster?
>>>
>>> Thanks,
>>> --
>>>
>>> *Tao Yan*
>>> Software Engineer
>>> Data Analytics Infrastructure Tools and Services
>>>
>>>
>>>
>>> 206.250.5345
>>> tyan@linkedin.com
>>> https://www.linkedin.com/in/taousc
>>>
>>
>>
>> _____________________________________________________________
>> The information contained in this communication is intended solely for
>> the use of the individual or entity to whom it is addressed and others
>> authorized to receive it. It may contain confidential or legally privileged
>> information. If you are not the intended recipient you are hereby notified
>> that any disclosure, copying, distribution or taking any action in reliance
>> on the contents of this information is strictly prohibited and may be
>> unlawful. If you have received this communication in error, please notify
>> us immediately by responding to this email and then delete it from your
>> system. The firm is neither liable for the proper and complete transmission
>> of the information contained in this communication nor for any delay in its
>> receipt.
>
>
>
>
> --
>
> *Tao Yan*
> Software Engineer
> Data Analytics Infrastructure Tools and Services
>
>
>
> 206.250.5345
> tyan@linkedin.com
> https://www.linkedin.com/in/taousc
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Lens Scalibility
Posted by Tao Yan <ty...@linkedin.com>.
Hi Puneet,
Thanks very much for the information. Do you mind sharing some information
about the number of engineers developing / maintaining Lens? the number of
users and the percentage time developers used to support them?
Thanks,
On Tue, Aug 30, 2016 at 1:18 AM, Puneet Gupta <pu...@inmobi.com>
wrote:
> Hi Tao
>
> You can find some discussion on this aspect in LENS-658
> <https://issues.apache.org/jira/browse/LENS-658>.
>
> As of now Lens server does not support clustered setup. From what we have
> seen in production, lens server consumes very little system resources as
> all the heavy lifting is done by target systems (Hive, Mysql, Vertica). The
> only operation on lens server that can consume some resources is query
> result formatting. We are able to easily support ~750 unique users and ~100
> concurrent users. We are running about 8000 queries per week.
>
> What is the kind or load your are expecting for your deployment ?
>
> Thanks,
> Puneet Gupta
>
> On Tue, Aug 30, 2016 at 6:18 AM, Tao Yan <ty...@linkedin.com> wrote:
>
>> Hi Lens Developers,
>>
>> As the number of users grow, one Lens server may not be able to support
>> all users. Is it possible to setup a multi-server Lens cluster?
>>
>> Thanks,
>> --
>>
>> *Tao Yan*
>> Software Engineer
>> Data Analytics Infrastructure Tools and Services
>>
>>
>>
>> 206.250.5345
>> tyan@linkedin.com
>> https://www.linkedin.com/in/taousc
>>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
--
*Tao Yan*
Software Engineer
Data Analytics Infrastructure Tools and Services
206.250.5345
tyan@linkedin.com
https://www.linkedin.com/in/taousc
Re: Lens Scalibility
Posted by Puneet Gupta <pu...@inmobi.com>.
Hi Tao
You can find some discussion on this aspect in LENS-658
<https://issues.apache.org/jira/browse/LENS-658>.
As of now Lens server does not support clustered setup. From what we have
seen in production, lens server consumes very little system resources as
all the heavy lifting is done by target systems (Hive, Mysql, Vertica). The
only operation on lens server that can consume some resources is query
result formatting. We are able to easily support ~750 unique users and ~100
concurrent users. We are running about 8000 queries per week.
What is the kind or load your are expecting for your deployment ?
Thanks,
Puneet Gupta
On Tue, Aug 30, 2016 at 6:18 AM, Tao Yan <ty...@linkedin.com> wrote:
> Hi Lens Developers,
>
> As the number of users grow, one Lens server may not be able to support
> all users. Is it possible to setup a multi-server Lens cluster?
>
> Thanks,
> --
>
> *Tao Yan*
> Software Engineer
> Data Analytics Infrastructure Tools and Services
>
>
>
> 206.250.5345
> tyan@linkedin.com
> https://www.linkedin.com/in/taousc
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.