You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "sunfl@certusnet.com.cn" <su...@certusnet.com.cn> on 2015/01/07 09:17:43 UTC
Re: Re: Fwd: Phoenix in production
Hi,
spark-phoenix integration would be great as Spark community is greately active now and more
and more developers are using Apache Spark.
Thanks,
Sun.
From: James Taylor
Date: 2015-01-07 16:10
To: sunfl@certusnet.com.cn
Subject: Re: Fwd: Phoenix in production
This is great, Sun! Thank you so much. Would you mind posting this on our user list in response to Siddharth's email? I think other Phoenix users would find it interesting as well.
On a side note, not sure how general what you developed is, but it would be interesting to pursue a general Spark integration in Phoenix as an open source contribution.
Thanks,
James
On Tue, Jan 6, 2015 at 5:41 PM, sunfl@certusnet.com.cn <su...@certusnet.com.cn> wrote:
Hi, James & Siddharth
Glad to share our experience of using Phoenix in Production. I believe that Siddharth had done
sufficient tests and practices about Phoenix performance. Here are some tips about how we are using
Phoenix for our projects:
1. We facilitate Phoenix to give convinience for both RD and QA engineers, as they are glad to use
standard sql to operate hbase with no much loss of query performance.
2. In Production environment, we mainly integrate Apache Spark with Phoenix to optimize data loading to
Phoenix tables with or withour secondary indexes. Glad that current performance of writing had worked
smoothly with both compared to previously used MySQL InfoBright and other sql schema. We both had tested
a lot for secondary indexes and query optimization for Phoenix before moving Phoenix to Production
environment. Now we can get most of the features worked for Phoenix in our job.
3. Challenges had a lot too, such as bulkload performance with wal enabled, query optimization, statistical data collection with Phoenix
full table scan, and so on. However, we believe Phoenix be a sufficient schema for sql query over HBase and we are glad
that even more our projects are considering using Phoenix.
Thanks,
Sun.
CertusNet
From: James Taylor
Date: 2015-01-07 09:10
To: sunfl@certusnet.com.cn
Subject: Fwd: Phoenix in production
Hi Sun,
Any experiences you can share with Siddharth?
Thanks,
James
---------- Forwarded message ----------
From: Siddharth Ubale <si...@syncoms.com>
Date: Thu, Jan 1, 2015 at 11:21 PM
Subject: Phoenix in production
To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
Hi Guys,
We are seriously thinking of phoenix in Production environment , however, we have no much data of how Phoenix is behaving in production.
Can anyone let us know if anyone is using Phoenix in Production and any challenges which they have experienced.
Thanks,
Siddharth Ubale,
Synchronized Communications
#43, Velankani Tech Park, Block No. II,
3rd Floor, Electronic City Phase I,
Bangalore – 560 100
Tel : +91 80 3202 4060
Web: www.syncoms.com
London|Bangalore|Orlando
we innovate, plan, execute, and transform the business
Re: Re: Fwd: Phoenix in production
Posted by Kristoffer Sjögren <st...@gmail.com>.
We have been using Phoenix 2.2.3 in production for about a year and I agree
with the previous comments.
- Mainly a storage for temporal OLAP-like data in single tables without
secondary indexes.
- Data ingested via pig on hourly basis.
- Heavy usage of composite primary keys using skip-scans whenever possible.
- Use salting.
- Jooq's MySQL dialect works quite well with Phoenix.
- The only real downtime we had was when we mistakenly connected to Phoenix
using a newer client which automatically upgraded the cluster. The forced
us to upgrade all region servers with new phoenix jar files.
- We are going to upgrade to Phoenix 3.0 sometime soon.
All and all happy with Phoenix. Just make sure to read the documentation
and follow advice found on http://phoenix.apache.org and you'll be fine.
Cheers,
-Kristoffer
On Wed, Jan 7, 2015 at 9:17 AM, sunfl@certusnet.com.cn <
sunfl@certusnet.com.cn> wrote:
> Hi,
> spark-phoenix integration would be great as Spark community is greately
> active now and more
> and more developers are using Apache Spark.
>
> Thanks,
> Sun.
>
> ------------------------------
> ------------------------------
>
>
> *From:* James Taylor <ja...@apache.org>
> *Date:* 2015-01-07 16:10
> *To:* sunfl@certusnet.com.cn
> *Subject:* Re: Fwd: Phoenix in production
> This is great, Sun! Thank you so much. Would you mind posting this on our
> user list in response to Siddharth's email? I think other Phoenix users
> would find it interesting as well.
>
> On a side note, not sure how general what you developed is, but it would
> be interesting to pursue a general Spark integration in Phoenix as an open
> source contribution.
>
> Thanks,
> James
>
> On Tue, Jan 6, 2015 at 5:41 PM, sunfl@certusnet.com.cn <
> sunfl@certusnet.com.cn> wrote:
>
>> Hi, James & Siddharth
>>
>> Glad to share our experience of using Phoenix in Production. I believe
>> that Siddharth had done
>> sufficient tests and practices about Phoenix performance. Here are some
>> tips about how we are using
>> Phoenix for our projects:
>> 1. We facilitate Phoenix to give convinience for both RD and QA
>> engineers, as they are glad to use
>> standard sql to operate hbase with no much loss of query performance.
>> 2. In Production environment, we mainly integrate Apache Spark with
>> Phoenix to optimize data loading to
>> Phoenix tables with or withour secondary indexes. Glad that current
>> performance of writing had worked
>> smoothly with both compared to previously used MySQL InfoBright and other
>> sql schema. We both had tested
>> a lot for secondary indexes and query optimization for Phoenix before
>> moving Phoenix to Production
>> environment. Now we can get most of the features worked for Phoenix in
>> our job.
>> 3. Challenges had a lot too, such as bulkload performance with wal
>> enabled, query optimization, statistical data collection with Phoenix
>> full table scan, and so on. However, we believe Phoenix be a sufficient
>> schema for sql query over HBase and we are glad
>> that even more our projects are considering using Phoenix.
>>
>> Thanks,
>> Sun.
>>
>> ------------------------------
>> ------------------------------
>>
>> CertusNet
>>
>>
>> *From:* James Taylor <ja...@apache.org>
>> *Date:* 2015-01-07 09:10
>> *To:* sunfl@certusnet.com.cn
>> *Subject:* Fwd: Phoenix in production
>> Hi Sun,
>> Any experiences you can share with Siddharth?
>> Thanks,
>> James
>>
>> ---------- Forwarded message ----------
>> From: Siddharth Ubale <si...@syncoms.com>
>> Date: Thu, Jan 1, 2015 at 11:21 PM
>> Subject: Phoenix in production
>> To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
>>
>>
>> Hi Guys,
>>
>>
>>
>> We are seriously thinking of phoenix in Production environment , however,
>> we have no much data of how Phoenix is behaving in production.
>>
>> Can anyone let us know if anyone is using Phoenix in Production and any
>> challenges which they have experienced.
>>
>>
>>
>> Thanks,
>>
>> Siddharth Ubale,
>>
>> *Synchronized Communications *
>>
>> *#43, Velankani Tech Park, Block No. II, *
>>
>> *3rd Floor, Electronic City Phase I,*
>>
>> *Bangalore – 560 100*
>>
>> *Tel : +91 80 3202 4060 <%2B91%2080%203202%204060>*
>>
>> *Web:* *www.syncoms.com* <http://www.syncoms.com/>
>>
>> *[image: LogoNEWmohLARGE]*
>>
>> *London*|*Bangalore*|*Orlando*
>>
>>
>>
>> *we innovate, plan, execute, and transform the business*
>>
>>
>>
>>
>
Re: Re: Fwd: Phoenix in production
Posted by Kristoffer Sjögren <st...@gmail.com>.
We have been using Phoenix 2.2.3 in production for about a year and I agree
with the previous comments.
- Mainly a storage for temporal OLAP-like data in single tables without
secondary indexes.
- Data ingested via pig on hourly basis.
- Heavy usage of composite primary keys using skip-scans whenever possible.
- Use salting.
- Jooq's MySQL dialect works quite well with Phoenix.
- The only real downtime we had was when we mistakenly connected to Phoenix
using a newer client which automatically upgraded the cluster. The forced
us to upgrade all region servers with new phoenix jar files.
- We are going to upgrade to Phoenix 3.0 sometime soon.
All and all happy with Phoenix. Just make sure to read the documentation
and follow advice found on http://phoenix.apache.org and you'll be fine.
Cheers,
-Kristoffer
On Wed, Jan 7, 2015 at 9:17 AM, sunfl@certusnet.com.cn <
sunfl@certusnet.com.cn> wrote:
> Hi,
> spark-phoenix integration would be great as Spark community is greately
> active now and more
> and more developers are using Apache Spark.
>
> Thanks,
> Sun.
>
> ------------------------------
> ------------------------------
>
>
> *From:* James Taylor <ja...@apache.org>
> *Date:* 2015-01-07 16:10
> *To:* sunfl@certusnet.com.cn
> *Subject:* Re: Fwd: Phoenix in production
> This is great, Sun! Thank you so much. Would you mind posting this on our
> user list in response to Siddharth's email? I think other Phoenix users
> would find it interesting as well.
>
> On a side note, not sure how general what you developed is, but it would
> be interesting to pursue a general Spark integration in Phoenix as an open
> source contribution.
>
> Thanks,
> James
>
> On Tue, Jan 6, 2015 at 5:41 PM, sunfl@certusnet.com.cn <
> sunfl@certusnet.com.cn> wrote:
>
>> Hi, James & Siddharth
>>
>> Glad to share our experience of using Phoenix in Production. I believe
>> that Siddharth had done
>> sufficient tests and practices about Phoenix performance. Here are some
>> tips about how we are using
>> Phoenix for our projects:
>> 1. We facilitate Phoenix to give convinience for both RD and QA
>> engineers, as they are glad to use
>> standard sql to operate hbase with no much loss of query performance.
>> 2. In Production environment, we mainly integrate Apache Spark with
>> Phoenix to optimize data loading to
>> Phoenix tables with or withour secondary indexes. Glad that current
>> performance of writing had worked
>> smoothly with both compared to previously used MySQL InfoBright and other
>> sql schema. We both had tested
>> a lot for secondary indexes and query optimization for Phoenix before
>> moving Phoenix to Production
>> environment. Now we can get most of the features worked for Phoenix in
>> our job.
>> 3. Challenges had a lot too, such as bulkload performance with wal
>> enabled, query optimization, statistical data collection with Phoenix
>> full table scan, and so on. However, we believe Phoenix be a sufficient
>> schema for sql query over HBase and we are glad
>> that even more our projects are considering using Phoenix.
>>
>> Thanks,
>> Sun.
>>
>> ------------------------------
>> ------------------------------
>>
>> CertusNet
>>
>>
>> *From:* James Taylor <ja...@apache.org>
>> *Date:* 2015-01-07 09:10
>> *To:* sunfl@certusnet.com.cn
>> *Subject:* Fwd: Phoenix in production
>> Hi Sun,
>> Any experiences you can share with Siddharth?
>> Thanks,
>> James
>>
>> ---------- Forwarded message ----------
>> From: Siddharth Ubale <si...@syncoms.com>
>> Date: Thu, Jan 1, 2015 at 11:21 PM
>> Subject: Phoenix in production
>> To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
>>
>>
>> Hi Guys,
>>
>>
>>
>> We are seriously thinking of phoenix in Production environment , however,
>> we have no much data of how Phoenix is behaving in production.
>>
>> Can anyone let us know if anyone is using Phoenix in Production and any
>> challenges which they have experienced.
>>
>>
>>
>> Thanks,
>>
>> Siddharth Ubale,
>>
>> *Synchronized Communications *
>>
>> *#43, Velankani Tech Park, Block No. II, *
>>
>> *3rd Floor, Electronic City Phase I,*
>>
>> *Bangalore – 560 100*
>>
>> *Tel : +91 80 3202 4060 <%2B91%2080%203202%204060>*
>>
>> *Web:* *www.syncoms.com* <http://www.syncoms.com/>
>>
>> *[image: LogoNEWmohLARGE]*
>>
>> *London*|*Bangalore*|*Orlando*
>>
>>
>>
>> *we innovate, plan, execute, and transform the business*
>>
>>
>>
>>
>
Re: Re: Fwd: Phoenix in production
Posted by Josh Mahonin <jm...@interset.com>.
On Wed, Jan 7, 2015 at 1:43 PM, anil gupta <an...@gmail.com> wrote:
> Yup, I am aware of Spark HBase integration. Phoenix-Spark integration
> would be more sweet. :)
Hi Anil,
I'm using Spark and Phoenix in production fairly successfully. There's very
little required for integration, since Phoenix has Hadoop Input and Output
formats that Spark can use natively.
As well, there is anther project which aims to bring the full Spark SQL
integration to Phoenix:
https://github.com/simplymeasured/phoenix-spark
Josh
Re: Re: Fwd: Phoenix in production
Posted by anil gupta <an...@gmail.com>.
Yup, I am aware of Spark HBase integration. Phoenix-Spark integration would
be more sweet. :)
On Wed, Jan 7, 2015 at 12:40 AM, sunfl@certusnet.com.cn <
sunfl@certusnet.com.cn> wrote:
> Hi Anil,
> Well, there are already good opensouce project on github for Spark on
> HBase, like the following:
> https://github.com/cloudera-labs/SparkOnHBase
>
> Phoenix integration shall be more convenient based on that. Considering to
> share our code for using
> that schema.
>
> Thanks,
> Sun.
>
> ------------------------------
> ------------------------------
>
> CertusNet
>
>
>
> *From:* anil gupta <an...@gmail.com>
> *Date:* 2015-01-07 16:28
> *To:* user@phoenix.apache.org
> *CC:* James Taylor <ja...@apache.org>; dev <de...@phoenix.apache.org>
> *Subject:* Re: Re: Fwd: Phoenix in production
> Hi Sun,
>
> Phoenix-Spark would be a nice addon if you can open source it. I am
> planning/thinking to using Spark on HBase for one of my project.
>
> ~Anil
>
> On Wed, Jan 7, 2015 at 12:17 AM, sunfl@certusnet.com.cn <
> sunfl@certusnet.com.cn> wrote:
>
>> Hi,
>> spark-phoenix integration would be great as Spark community is greately
>> active now and more
>> and more developers are using Apache Spark.
>>
>> Thanks,
>> Sun.
>>
>> ------------------------------
>> ------------------------------
>>
>>
>> *From:* James Taylor <ja...@apache.org>
>> *Date:* 2015-01-07 16:10
>> *To:* sunfl@certusnet.com.cn
>> *Subject:* Re: Fwd: Phoenix in production
>> This is great, Sun! Thank you so much. Would you mind posting this on our
>> user list in response to Siddharth's email? I think other Phoenix users
>> would find it interesting as well.
>>
>> On a side note, not sure how general what you developed is, but it would
>> be interesting to pursue a general Spark integration in Phoenix as an open
>> source contribution.
>>
>> Thanks,
>> James
>>
>> On Tue, Jan 6, 2015 at 5:41 PM, sunfl@certusnet.com.cn <
>> sunfl@certusnet.com.cn> wrote:
>>
>>> Hi, James & Siddharth
>>>
>>> Glad to share our experience of using Phoenix in Production. I believe
>>> that Siddharth had done
>>> sufficient tests and practices about Phoenix performance. Here are some
>>> tips about how we are using
>>> Phoenix for our projects:
>>> 1. We facilitate Phoenix to give convinience for both RD and QA
>>> engineers, as they are glad to use
>>> standard sql to operate hbase with no much loss of query performance.
>>> 2. In Production environment, we mainly integrate Apache Spark with
>>> Phoenix to optimize data loading to
>>> Phoenix tables with or withour secondary indexes. Glad that current
>>> performance of writing had worked
>>> smoothly with both compared to previously used MySQL InfoBright and
>>> other sql schema. We both had tested
>>> a lot for secondary indexes and query optimization for Phoenix before
>>> moving Phoenix to Production
>>> environment. Now we can get most of the features worked for Phoenix in
>>> our job.
>>> 3. Challenges had a lot too, such as bulkload performance with wal
>>> enabled, query optimization, statistical data collection with Phoenix
>>> full table scan, and so on. However, we believe Phoenix be a sufficient
>>> schema for sql query over HBase and we are glad
>>> that even more our projects are considering using Phoenix.
>>>
>>> Thanks,
>>> Sun.
>>>
>>> ------------------------------
>>> ------------------------------
>>>
>>> CertusNet
>>>
>>>
>>> *From:* James Taylor <ja...@apache.org>
>>> *Date:* 2015-01-07 09:10
>>> *To:* sunfl@certusnet.com.cn
>>> *Subject:* Fwd: Phoenix in production
>>> Hi Sun,
>>> Any experiences you can share with Siddharth?
>>> Thanks,
>>> James
>>>
>>> ---------- Forwarded message ----------
>>> From: Siddharth Ubale <si...@syncoms.com>
>>> Date: Thu, Jan 1, 2015 at 11:21 PM
>>> Subject: Phoenix in production
>>> To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
>>>
>>>
>>> Hi Guys,
>>>
>>>
>>>
>>> We are seriously thinking of phoenix in Production environment ,
>>> however, we have no much data of how Phoenix is behaving in production.
>>>
>>> Can anyone let us know if anyone is using Phoenix in Production and any
>>> challenges which they have experienced.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Siddharth Ubale,
>>>
>>> *Synchronized Communications *
>>>
>>> *#43, Velankani Tech Park, Block No. II, *
>>>
>>> *3rd Floor, Electronic City Phase I,*
>>>
>>> *Bangalore – 560 100*
>>>
>>> *Tel : +91 80 3202 4060 <%2B91%2080%203202%204060>*
>>>
>>> *Web:* *www.syncoms.com* <http://www.syncoms.com/>
>>>
>>> *[image: LogoNEWmohLARGE]*
>>>
>>> *London*|*Bangalore*|*Orlando*
>>>
>>>
>>>
>>> *we innovate, plan, execute, and transform the business*
>>>
>>>
>>>
>>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
>
--
Thanks & Regards,
Anil Gupta
Re: Re: Fwd: Phoenix in production
Posted by anil gupta <an...@gmail.com>.
Yup, I am aware of Spark HBase integration. Phoenix-Spark integration would
be more sweet. :)
On Wed, Jan 7, 2015 at 12:40 AM, sunfl@certusnet.com.cn <
sunfl@certusnet.com.cn> wrote:
> Hi Anil,
> Well, there are already good opensouce project on github for Spark on
> HBase, like the following:
> https://github.com/cloudera-labs/SparkOnHBase
>
> Phoenix integration shall be more convenient based on that. Considering to
> share our code for using
> that schema.
>
> Thanks,
> Sun.
>
> ------------------------------
> ------------------------------
>
> CertusNet
>
>
>
> *From:* anil gupta <an...@gmail.com>
> *Date:* 2015-01-07 16:28
> *To:* user@phoenix.apache.org
> *CC:* James Taylor <ja...@apache.org>; dev <de...@phoenix.apache.org>
> *Subject:* Re: Re: Fwd: Phoenix in production
> Hi Sun,
>
> Phoenix-Spark would be a nice addon if you can open source it. I am
> planning/thinking to using Spark on HBase for one of my project.
>
> ~Anil
>
> On Wed, Jan 7, 2015 at 12:17 AM, sunfl@certusnet.com.cn <
> sunfl@certusnet.com.cn> wrote:
>
>> Hi,
>> spark-phoenix integration would be great as Spark community is greately
>> active now and more
>> and more developers are using Apache Spark.
>>
>> Thanks,
>> Sun.
>>
>> ------------------------------
>> ------------------------------
>>
>>
>> *From:* James Taylor <ja...@apache.org>
>> *Date:* 2015-01-07 16:10
>> *To:* sunfl@certusnet.com.cn
>> *Subject:* Re: Fwd: Phoenix in production
>> This is great, Sun! Thank you so much. Would you mind posting this on our
>> user list in response to Siddharth's email? I think other Phoenix users
>> would find it interesting as well.
>>
>> On a side note, not sure how general what you developed is, but it would
>> be interesting to pursue a general Spark integration in Phoenix as an open
>> source contribution.
>>
>> Thanks,
>> James
>>
>> On Tue, Jan 6, 2015 at 5:41 PM, sunfl@certusnet.com.cn <
>> sunfl@certusnet.com.cn> wrote:
>>
>>> Hi, James & Siddharth
>>>
>>> Glad to share our experience of using Phoenix in Production. I believe
>>> that Siddharth had done
>>> sufficient tests and practices about Phoenix performance. Here are some
>>> tips about how we are using
>>> Phoenix for our projects:
>>> 1. We facilitate Phoenix to give convinience for both RD and QA
>>> engineers, as they are glad to use
>>> standard sql to operate hbase with no much loss of query performance.
>>> 2. In Production environment, we mainly integrate Apache Spark with
>>> Phoenix to optimize data loading to
>>> Phoenix tables with or withour secondary indexes. Glad that current
>>> performance of writing had worked
>>> smoothly with both compared to previously used MySQL InfoBright and
>>> other sql schema. We both had tested
>>> a lot for secondary indexes and query optimization for Phoenix before
>>> moving Phoenix to Production
>>> environment. Now we can get most of the features worked for Phoenix in
>>> our job.
>>> 3. Challenges had a lot too, such as bulkload performance with wal
>>> enabled, query optimization, statistical data collection with Phoenix
>>> full table scan, and so on. However, we believe Phoenix be a sufficient
>>> schema for sql query over HBase and we are glad
>>> that even more our projects are considering using Phoenix.
>>>
>>> Thanks,
>>> Sun.
>>>
>>> ------------------------------
>>> ------------------------------
>>>
>>> CertusNet
>>>
>>>
>>> *From:* James Taylor <ja...@apache.org>
>>> *Date:* 2015-01-07 09:10
>>> *To:* sunfl@certusnet.com.cn
>>> *Subject:* Fwd: Phoenix in production
>>> Hi Sun,
>>> Any experiences you can share with Siddharth?
>>> Thanks,
>>> James
>>>
>>> ---------- Forwarded message ----------
>>> From: Siddharth Ubale <si...@syncoms.com>
>>> Date: Thu, Jan 1, 2015 at 11:21 PM
>>> Subject: Phoenix in production
>>> To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
>>>
>>>
>>> Hi Guys,
>>>
>>>
>>>
>>> We are seriously thinking of phoenix in Production environment ,
>>> however, we have no much data of how Phoenix is behaving in production.
>>>
>>> Can anyone let us know if anyone is using Phoenix in Production and any
>>> challenges which they have experienced.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Siddharth Ubale,
>>>
>>> *Synchronized Communications *
>>>
>>> *#43, Velankani Tech Park, Block No. II, *
>>>
>>> *3rd Floor, Electronic City Phase I,*
>>>
>>> *Bangalore – 560 100*
>>>
>>> *Tel : +91 80 3202 4060 <%2B91%2080%203202%204060>*
>>>
>>> *Web:* *www.syncoms.com* <http://www.syncoms.com/>
>>>
>>> *[image: LogoNEWmohLARGE]*
>>>
>>> *London*|*Bangalore*|*Orlando*
>>>
>>>
>>>
>>> *we innovate, plan, execute, and transform the business*
>>>
>>>
>>>
>>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
>
--
Thanks & Regards,
Anil Gupta
Re: Re: Fwd: Phoenix in production
Posted by "sunfl@certusnet.com.cn" <su...@certusnet.com.cn>.
Hi Anil,
Well, there are already good opensouce project on github for Spark on HBase, like the following:
https://github.com/cloudera-labs/SparkOnHBase
Phoenix integration shall be more convenient based on that. Considering to share our code for using
that schema.
Thanks,
Sun.
CertusNet
From: anil gupta
Date: 2015-01-07 16:28
To: user@phoenix.apache.org
CC: James Taylor; dev
Subject: Re: Re: Fwd: Phoenix in production
Hi Sun,
Phoenix-Spark would be a nice addon if you can open source it. I am planning/thinking to using Spark on HBase for one of my project.
~Anil
On Wed, Jan 7, 2015 at 12:17 AM, sunfl@certusnet.com.cn <su...@certusnet.com.cn> wrote:
Hi,
spark-phoenix integration would be great as Spark community is greately active now and more
and more developers are using Apache Spark.
Thanks,
Sun.
From: James Taylor
Date: 2015-01-07 16:10
To: sunfl@certusnet.com.cn
Subject: Re: Fwd: Phoenix in production
This is great, Sun! Thank you so much. Would you mind posting this on our user list in response to Siddharth's email? I think other Phoenix users would find it interesting as well.
On a side note, not sure how general what you developed is, but it would be interesting to pursue a general Spark integration in Phoenix as an open source contribution.
Thanks,
James
On Tue, Jan 6, 2015 at 5:41 PM, sunfl@certusnet.com.cn <su...@certusnet.com.cn> wrote:
Hi, James & Siddharth
Glad to share our experience of using Phoenix in Production. I believe that Siddharth had done
sufficient tests and practices about Phoenix performance. Here are some tips about how we are using
Phoenix for our projects:
1. We facilitate Phoenix to give convinience for both RD and QA engineers, as they are glad to use
standard sql to operate hbase with no much loss of query performance.
2. In Production environment, we mainly integrate Apache Spark with Phoenix to optimize data loading to
Phoenix tables with or withour secondary indexes. Glad that current performance of writing had worked
smoothly with both compared to previously used MySQL InfoBright and other sql schema. We both had tested
a lot for secondary indexes and query optimization for Phoenix before moving Phoenix to Production
environment. Now we can get most of the features worked for Phoenix in our job.
3. Challenges had a lot too, such as bulkload performance with wal enabled, query optimization, statistical data collection with Phoenix
full table scan, and so on. However, we believe Phoenix be a sufficient schema for sql query over HBase and we are glad
that even more our projects are considering using Phoenix.
Thanks,
Sun.
CertusNet
From: James Taylor
Date: 2015-01-07 09:10
To: sunfl@certusnet.com.cn
Subject: Fwd: Phoenix in production
Hi Sun,
Any experiences you can share with Siddharth?
Thanks,
James
---------- Forwarded message ----------
From: Siddharth Ubale <si...@syncoms.com>
Date: Thu, Jan 1, 2015 at 11:21 PM
Subject: Phoenix in production
To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
Hi Guys,
We are seriously thinking of phoenix in Production environment , however, we have no much data of how Phoenix is behaving in production.
Can anyone let us know if anyone is using Phoenix in Production and any challenges which they have experienced.
Thanks,
Siddharth Ubale,
Synchronized Communications
#43, Velankani Tech Park, Block No. II,
3rd Floor, Electronic City Phase I,
Bangalore – 560 100
Tel : +91 80 3202 4060
Web: www.syncoms.com
London|Bangalore|Orlando
we innovate, plan, execute, and transform the business
--
Thanks & Regards,
Anil Gupta
Re: Re: Fwd: Phoenix in production
Posted by "sunfl@certusnet.com.cn" <su...@certusnet.com.cn>.
Hi Anil,
Well, there are already good opensouce project on github for Spark on HBase, like the following:
https://github.com/cloudera-labs/SparkOnHBase
Phoenix integration shall be more convenient based on that. Considering to share our code for using
that schema.
Thanks,
Sun.
CertusNet
From: anil gupta
Date: 2015-01-07 16:28
To: user@phoenix.apache.org
CC: James Taylor; dev
Subject: Re: Re: Fwd: Phoenix in production
Hi Sun,
Phoenix-Spark would be a nice addon if you can open source it. I am planning/thinking to using Spark on HBase for one of my project.
~Anil
On Wed, Jan 7, 2015 at 12:17 AM, sunfl@certusnet.com.cn <su...@certusnet.com.cn> wrote:
Hi,
spark-phoenix integration would be great as Spark community is greately active now and more
and more developers are using Apache Spark.
Thanks,
Sun.
From: James Taylor
Date: 2015-01-07 16:10
To: sunfl@certusnet.com.cn
Subject: Re: Fwd: Phoenix in production
This is great, Sun! Thank you so much. Would you mind posting this on our user list in response to Siddharth's email? I think other Phoenix users would find it interesting as well.
On a side note, not sure how general what you developed is, but it would be interesting to pursue a general Spark integration in Phoenix as an open source contribution.
Thanks,
James
On Tue, Jan 6, 2015 at 5:41 PM, sunfl@certusnet.com.cn <su...@certusnet.com.cn> wrote:
Hi, James & Siddharth
Glad to share our experience of using Phoenix in Production. I believe that Siddharth had done
sufficient tests and practices about Phoenix performance. Here are some tips about how we are using
Phoenix for our projects:
1. We facilitate Phoenix to give convinience for both RD and QA engineers, as they are glad to use
standard sql to operate hbase with no much loss of query performance.
2. In Production environment, we mainly integrate Apache Spark with Phoenix to optimize data loading to
Phoenix tables with or withour secondary indexes. Glad that current performance of writing had worked
smoothly with both compared to previously used MySQL InfoBright and other sql schema. We both had tested
a lot for secondary indexes and query optimization for Phoenix before moving Phoenix to Production
environment. Now we can get most of the features worked for Phoenix in our job.
3. Challenges had a lot too, such as bulkload performance with wal enabled, query optimization, statistical data collection with Phoenix
full table scan, and so on. However, we believe Phoenix be a sufficient schema for sql query over HBase and we are glad
that even more our projects are considering using Phoenix.
Thanks,
Sun.
CertusNet
From: James Taylor
Date: 2015-01-07 09:10
To: sunfl@certusnet.com.cn
Subject: Fwd: Phoenix in production
Hi Sun,
Any experiences you can share with Siddharth?
Thanks,
James
---------- Forwarded message ----------
From: Siddharth Ubale <si...@syncoms.com>
Date: Thu, Jan 1, 2015 at 11:21 PM
Subject: Phoenix in production
To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
Hi Guys,
We are seriously thinking of phoenix in Production environment , however, we have no much data of how Phoenix is behaving in production.
Can anyone let us know if anyone is using Phoenix in Production and any challenges which they have experienced.
Thanks,
Siddharth Ubale,
Synchronized Communications
#43, Velankani Tech Park, Block No. II,
3rd Floor, Electronic City Phase I,
Bangalore – 560 100
Tel : +91 80 3202 4060
Web: www.syncoms.com
London|Bangalore|Orlando
we innovate, plan, execute, and transform the business
--
Thanks & Regards,
Anil Gupta
Re: Re: Fwd: Phoenix in production
Posted by anil gupta <an...@gmail.com>.
Hi Sun,
Phoenix-Spark would be a nice addon if you can open source it. I am
planning/thinking to using Spark on HBase for one of my project.
~Anil
On Wed, Jan 7, 2015 at 12:17 AM, sunfl@certusnet.com.cn <
sunfl@certusnet.com.cn> wrote:
> Hi,
> spark-phoenix integration would be great as Spark community is greately
> active now and more
> and more developers are using Apache Spark.
>
> Thanks,
> Sun.
>
> ------------------------------
> ------------------------------
>
>
> *From:* James Taylor <ja...@apache.org>
> *Date:* 2015-01-07 16:10
> *To:* sunfl@certusnet.com.cn
> *Subject:* Re: Fwd: Phoenix in production
> This is great, Sun! Thank you so much. Would you mind posting this on our
> user list in response to Siddharth's email? I think other Phoenix users
> would find it interesting as well.
>
> On a side note, not sure how general what you developed is, but it would
> be interesting to pursue a general Spark integration in Phoenix as an open
> source contribution.
>
> Thanks,
> James
>
> On Tue, Jan 6, 2015 at 5:41 PM, sunfl@certusnet.com.cn <
> sunfl@certusnet.com.cn> wrote:
>
>> Hi, James & Siddharth
>>
>> Glad to share our experience of using Phoenix in Production. I believe
>> that Siddharth had done
>> sufficient tests and practices about Phoenix performance. Here are some
>> tips about how we are using
>> Phoenix for our projects:
>> 1. We facilitate Phoenix to give convinience for both RD and QA
>> engineers, as they are glad to use
>> standard sql to operate hbase with no much loss of query performance.
>> 2. In Production environment, we mainly integrate Apache Spark with
>> Phoenix to optimize data loading to
>> Phoenix tables with or withour secondary indexes. Glad that current
>> performance of writing had worked
>> smoothly with both compared to previously used MySQL InfoBright and other
>> sql schema. We both had tested
>> a lot for secondary indexes and query optimization for Phoenix before
>> moving Phoenix to Production
>> environment. Now we can get most of the features worked for Phoenix in
>> our job.
>> 3. Challenges had a lot too, such as bulkload performance with wal
>> enabled, query optimization, statistical data collection with Phoenix
>> full table scan, and so on. However, we believe Phoenix be a sufficient
>> schema for sql query over HBase and we are glad
>> that even more our projects are considering using Phoenix.
>>
>> Thanks,
>> Sun.
>>
>> ------------------------------
>> ------------------------------
>>
>> CertusNet
>>
>>
>> *From:* James Taylor <ja...@apache.org>
>> *Date:* 2015-01-07 09:10
>> *To:* sunfl@certusnet.com.cn
>> *Subject:* Fwd: Phoenix in production
>> Hi Sun,
>> Any experiences you can share with Siddharth?
>> Thanks,
>> James
>>
>> ---------- Forwarded message ----------
>> From: Siddharth Ubale <si...@syncoms.com>
>> Date: Thu, Jan 1, 2015 at 11:21 PM
>> Subject: Phoenix in production
>> To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
>>
>>
>> Hi Guys,
>>
>>
>>
>> We are seriously thinking of phoenix in Production environment , however,
>> we have no much data of how Phoenix is behaving in production.
>>
>> Can anyone let us know if anyone is using Phoenix in Production and any
>> challenges which they have experienced.
>>
>>
>>
>> Thanks,
>>
>> Siddharth Ubale,
>>
>> *Synchronized Communications *
>>
>> *#43, Velankani Tech Park, Block No. II, *
>>
>> *3rd Floor, Electronic City Phase I,*
>>
>> *Bangalore – 560 100*
>>
>> *Tel : +91 80 3202 4060 <%2B91%2080%203202%204060>*
>>
>> *Web:* *www.syncoms.com* <http://www.syncoms.com/>
>>
>> *[image: LogoNEWmohLARGE]*
>>
>> *London*|*Bangalore*|*Orlando*
>>
>>
>>
>> *we innovate, plan, execute, and transform the business*
>>
>>
>>
>>
>
--
Thanks & Regards,
Anil Gupta
Re: Re: Fwd: Phoenix in production
Posted by anil gupta <an...@gmail.com>.
Hi Sun,
Phoenix-Spark would be a nice addon if you can open source it. I am
planning/thinking to using Spark on HBase for one of my project.
~Anil
On Wed, Jan 7, 2015 at 12:17 AM, sunfl@certusnet.com.cn <
sunfl@certusnet.com.cn> wrote:
> Hi,
> spark-phoenix integration would be great as Spark community is greately
> active now and more
> and more developers are using Apache Spark.
>
> Thanks,
> Sun.
>
> ------------------------------
> ------------------------------
>
>
> *From:* James Taylor <ja...@apache.org>
> *Date:* 2015-01-07 16:10
> *To:* sunfl@certusnet.com.cn
> *Subject:* Re: Fwd: Phoenix in production
> This is great, Sun! Thank you so much. Would you mind posting this on our
> user list in response to Siddharth's email? I think other Phoenix users
> would find it interesting as well.
>
> On a side note, not sure how general what you developed is, but it would
> be interesting to pursue a general Spark integration in Phoenix as an open
> source contribution.
>
> Thanks,
> James
>
> On Tue, Jan 6, 2015 at 5:41 PM, sunfl@certusnet.com.cn <
> sunfl@certusnet.com.cn> wrote:
>
>> Hi, James & Siddharth
>>
>> Glad to share our experience of using Phoenix in Production. I believe
>> that Siddharth had done
>> sufficient tests and practices about Phoenix performance. Here are some
>> tips about how we are using
>> Phoenix for our projects:
>> 1. We facilitate Phoenix to give convinience for both RD and QA
>> engineers, as they are glad to use
>> standard sql to operate hbase with no much loss of query performance.
>> 2. In Production environment, we mainly integrate Apache Spark with
>> Phoenix to optimize data loading to
>> Phoenix tables with or withour secondary indexes. Glad that current
>> performance of writing had worked
>> smoothly with both compared to previously used MySQL InfoBright and other
>> sql schema. We both had tested
>> a lot for secondary indexes and query optimization for Phoenix before
>> moving Phoenix to Production
>> environment. Now we can get most of the features worked for Phoenix in
>> our job.
>> 3. Challenges had a lot too, such as bulkload performance with wal
>> enabled, query optimization, statistical data collection with Phoenix
>> full table scan, and so on. However, we believe Phoenix be a sufficient
>> schema for sql query over HBase and we are glad
>> that even more our projects are considering using Phoenix.
>>
>> Thanks,
>> Sun.
>>
>> ------------------------------
>> ------------------------------
>>
>> CertusNet
>>
>>
>> *From:* James Taylor <ja...@apache.org>
>> *Date:* 2015-01-07 09:10
>> *To:* sunfl@certusnet.com.cn
>> *Subject:* Fwd: Phoenix in production
>> Hi Sun,
>> Any experiences you can share with Siddharth?
>> Thanks,
>> James
>>
>> ---------- Forwarded message ----------
>> From: Siddharth Ubale <si...@syncoms.com>
>> Date: Thu, Jan 1, 2015 at 11:21 PM
>> Subject: Phoenix in production
>> To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
>>
>>
>> Hi Guys,
>>
>>
>>
>> We are seriously thinking of phoenix in Production environment , however,
>> we have no much data of how Phoenix is behaving in production.
>>
>> Can anyone let us know if anyone is using Phoenix in Production and any
>> challenges which they have experienced.
>>
>>
>>
>> Thanks,
>>
>> Siddharth Ubale,
>>
>> *Synchronized Communications *
>>
>> *#43, Velankani Tech Park, Block No. II, *
>>
>> *3rd Floor, Electronic City Phase I,*
>>
>> *Bangalore – 560 100*
>>
>> *Tel : +91 80 3202 4060 <%2B91%2080%203202%204060>*
>>
>> *Web:* *www.syncoms.com* <http://www.syncoms.com/>
>>
>> *[image: LogoNEWmohLARGE]*
>>
>> *London*|*Bangalore*|*Orlando*
>>
>>
>>
>> *we innovate, plan, execute, and transform the business*
>>
>>
>>
>>
>
--
Thanks & Regards,
Anil Gupta