You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Localhost shell <un...@gmail.com> on 2014/05/05 23:42:13 UTC
Inserting data in Hbase Phoenix using the thrift api
Hey Folks,
I have a use case where one of the apps(C++ server) will pump data into the
Hbase.
Since Phoenix doesn't support ODBC api's so the app will not be able to use
the Phoenix JDBC api and will use Hbase thirft api to insert the data.
Note: The app that is inserting data will create the row keys similar to
the way how Phoenix JDBC creates.
Currently no data resides in Hbase and the table will be freshly created
using the SQL commands (using phoenix sqlline).
All the analysis/group-by queries will be triggered by a different app
using the Phoenix JDBC API's.
In the above mention scenario, Are there any Phoenix functionalities ( for
ex: Salting, Secondary indexing) that will not be available because Phoenix
JDBC driver is not used for inserting data.
Can someone please share their thoughts on this?
Hadoop Distro: CDH5
HBase: 0.96.1
--Unilocal
Re: Inserting data in Hbase Phoenix using the thrift api
Posted by Localhost shell <un...@gmail.com>.
Hello All,
I am not sure if this mail went across as usually I have seen replies from
phoenix community.
Can someone please share their thoughts on this?
--Unilocal
On Mon, May 5, 2014 at 2:42 PM, Localhost shell <
universal.localhost@gmail.com> wrote:
> Hey Folks,
>
> I have a use case where one of the apps(C++ server) will pump data into
> the Hbase.
> Since Phoenix doesn't support ODBC api's so the app will not be able to
> use the Phoenix JDBC api and will use Hbase thirft api to insert the data.
> Note: The app that is inserting data will create the row keys similar to
> the way how Phoenix JDBC creates.
>
> Currently no data resides in Hbase and the table will be freshly created
> using the SQL commands (using phoenix sqlline).
> All the analysis/group-by queries will be triggered by a different app
> using the Phoenix JDBC API's.
>
> In the above mention scenario, Are there any Phoenix functionalities ( for
> ex: Salting, Secondary indexing) that will not be available because Phoenix
> JDBC driver is not used for inserting data.
>
> Can someone please share their thoughts on this?
>
> Hadoop Distro: CDH5
> HBase: 0.96.1
>
> --Unilocal
>
>
>
Re: Inserting data in Hbase Phoenix using the thrift api
Posted by alex kamil <al...@gmail.com>.
..and you can use something like
Avro<http://avro.apache.org/docs/1.7.6/api/cpp/html/index.html>or
ProtoBuf
<https://developers.google.com/protocol-buffers/docs/cpptutorial>for
serialization
On Sat, May 10, 2014 at 11:33 PM, alex kamil <al...@gmail.com> wrote:
> Unilocal,
>
> you can also decouple hbase from your app by streaming data via a simple
> unix pipe, unix socket or tcp socket,
> just create a socket listener in Java on hbase side (which uses phoenix
> JDBC to insert data into hbase)
>
> Alex
>
>
> On Thu, May 8, 2014 at 2:31 AM, James Taylor <ja...@apache.org>wrote:
>
>> Hi Unilocal,
>> Yes, both salting and secondary indexing rely on the Phoenix client in
>> cooperation with the server.
>>
>> Would it be possible for the C++ server to generate CSV files instead?
>> Then these could be pumped into Phoenix through our CSV bulk loader (which
>> could potentially be invoked through a variety of ways). Another
>> alternative may be through our Apache Pig integration. Or it'd be pretty
>> easy to adapt our Pig store func to a Hive SerDe. Then you could use the
>> Hive ODBC driver to pump in data that's formated in a Phoenix compliant
>> manner.
>>
>> If none of these are options, you could pump into a Phoenix table and
>> then transfer the data (using Phoenix APIs) through UPSERT SELECT into a
>> salted table or a table with secondary indexes.
>>
>> Thanks,
>> James
>>
>>
>> On Mon, May 5, 2014 at 2:42 PM, Localhost shell <
>> universal.localhost@gmail.com> wrote:
>>
>>> Hey Folks,
>>>
>>> I have a use case where one of the apps(C++ server) will pump data into
>>> the Hbase.
>>> Since Phoenix doesn't support ODBC api's so the app will not be able to
>>> use the Phoenix JDBC api and will use Hbase thirft api to insert the data.
>>> Note: The app that is inserting data will create the row keys similar to
>>> the way how Phoenix JDBC creates.
>>>
>>> Currently no data resides in Hbase and the table will be freshly created
>>> using the SQL commands (using phoenix sqlline).
>>> All the analysis/group-by queries will be triggered by a different app
>>> using the Phoenix JDBC API's.
>>>
>>> In the above mention scenario, Are there any Phoenix functionalities (
>>> for ex: Salting, Secondary indexing) that will not be available because
>>> Phoenix JDBC driver is not used for inserting data.
>>>
>>> Can someone please share their thoughts on this?
>>>
>>> Hadoop Distro: CDH5
>>> HBase: 0.96.1
>>>
>>> --Unilocal
>>>
>>>
>>>
>>
>>
>
Re: Inserting data in Hbase Phoenix using the thrift api
Posted by alex kamil <al...@gmail.com>.
Unilocal,
you can also decouple hbase from your app by streaming data via a simple
unix pipe, unix socket or tcp socket,
just create a socket listener in Java on hbase side (which uses phoenix
JDBC to insert data into hbase)
Alex
On Thu, May 8, 2014 at 2:31 AM, James Taylor <ja...@apache.org> wrote:
> Hi Unilocal,
> Yes, both salting and secondary indexing rely on the Phoenix client in
> cooperation with the server.
>
> Would it be possible for the C++ server to generate CSV files instead?
> Then these could be pumped into Phoenix through our CSV bulk loader (which
> could potentially be invoked through a variety of ways). Another
> alternative may be through our Apache Pig integration. Or it'd be pretty
> easy to adapt our Pig store func to a Hive SerDe. Then you could use the
> Hive ODBC driver to pump in data that's formated in a Phoenix compliant
> manner.
>
> If none of these are options, you could pump into a Phoenix table and then
> transfer the data (using Phoenix APIs) through UPSERT SELECT into a salted
> table or a table with secondary indexes.
>
> Thanks,
> James
>
>
> On Mon, May 5, 2014 at 2:42 PM, Localhost shell <
> universal.localhost@gmail.com> wrote:
>
>> Hey Folks,
>>
>> I have a use case where one of the apps(C++ server) will pump data into
>> the Hbase.
>> Since Phoenix doesn't support ODBC api's so the app will not be able to
>> use the Phoenix JDBC api and will use Hbase thirft api to insert the data.
>> Note: The app that is inserting data will create the row keys similar to
>> the way how Phoenix JDBC creates.
>>
>> Currently no data resides in Hbase and the table will be freshly created
>> using the SQL commands (using phoenix sqlline).
>> All the analysis/group-by queries will be triggered by a different app
>> using the Phoenix JDBC API's.
>>
>> In the above mention scenario, Are there any Phoenix functionalities (
>> for ex: Salting, Secondary indexing) that will not be available because
>> Phoenix JDBC driver is not used for inserting data.
>>
>> Can someone please share their thoughts on this?
>>
>> Hadoop Distro: CDH5
>> HBase: 0.96.1
>>
>> --Unilocal
>>
>>
>>
>
>
Re: Inserting data in Hbase Phoenix using the thrift api
Posted by Stan Campbell <st...@gmail.com>.
Now *that* is an interesting hack.
On Tue, May 13, 2014 at 3:05 PM, Harshit Bapna <hr...@gmail.com> wrote:
> Hey Alex,
>
> Thanks for creating the ticket.
> I am currently discussing this internally and will need your help if we
> decide to take this approach.
>
> Another approach suggested by Jeffrey: Hack HBase thrift server to use
> Phoenix instead of Hbase native api.
>
>
>
>
> On Sun, May 11, 2014 at 1:40 AM, alex kamil <al...@gmail.com> wrote:
>
>> I opened a ticket: https://issues.apache.org/jira/browse/PHOENIX-974 ,if it works out for you, contributions welcome)
>> we use kafka as a buffer between app and db, it supports pretty high
>> throughput <https://kafka.apache.org/07/performance.html> but may be not
>> be suitable for very latency sensitive writes, your best bet is direct
>> socket connection or some kind of distributed cache
>>
>>
>>
>> On Sun, May 11, 2014 at 4:06 AM, universal localhost <
>> universal.localhost@gmail.com> wrote:
>>
>>> Thanks a lot James & Alex for your answer n suggestions.
>>>
>>> So *apart from Salting and Secondary Indexing, **are there any other
>>> functionalities* *that I will loose by not using the phoenix JDBC api's
>>> for inserting data*.
>>> I think the above one is similar to 'Map to an existing HBase table'
>>> case.
>>>
>>> The last option that u suggested rdg Hive ODBC driver sounds
>>> interesting but I need to think a bit more on what it means :)
>>> Rest all the options that you have suggested are nice but less suitable
>>> for me as we aim to have a low lag between when the txn is first seen and
>>> when its ready for analysis.
>>>
>>>
>>> Alex, I didn't think about this approach. The C++ app server handles
>>> close 50-60 tns per sec so I was a bit cautious but definitely worth a try.
>>> Thnxs.
>>>
>>>
>>> --Unilocal
>>>
>>>
>>> On Wed, May 7, 2014 at 11:31 PM, James Taylor <ja...@apache.org>wrote:
>>>
>>>> Hi Unilocal,
>>>> Yes, both salting and secondary indexing rely on the Phoenix client in
>>>> cooperation with the server.
>>>>
>>>> Would it be possible for the C++ server to generate CSV files instead?
>>>> Then these could be pumped into Phoenix through our CSV bulk loader (which
>>>> could potentially be invoked through a variety of ways). Another
>>>> alternative may be through our Apache Pig integration. Or it'd be pretty
>>>> easy to adapt our Pig store func to a Hive SerDe. Then you could use the
>>>> Hive ODBC driver to pump in data that's formated in a Phoenix compliant
>>>> manner.
>>>>
>>>> If none of these are options, you could pump into a Phoenix table and
>>>> then transfer the data (using Phoenix APIs) through UPSERT SELECT into a
>>>> salted table or a table with secondary indexes.
>>>>
>>>> Thanks,
>>>> James
>>>>
>>>>
>>>> On Mon, May 5, 2014 at 2:42 PM, Localhost shell <
>>>> universal.localhost@gmail.com> wrote:
>>>>
>>>>> Hey Folks,
>>>>>
>>>>> I have a use case where one of the apps(C++ server) will pump data
>>>>> into the Hbase.
>>>>> Since Phoenix doesn't support ODBC api's so the app will not be able
>>>>> to use the Phoenix JDBC api and will use Hbase thirft api to insert the
>>>>> data. Note: The app that is inserting data will create the row keys similar
>>>>> to the way how Phoenix JDBC creates.
>>>>>
>>>>> Currently no data resides in Hbase and the table will be freshly
>>>>> created using the SQL commands (using phoenix sqlline).
>>>>> All the analysis/group-by queries will be triggered by a different app
>>>>> using the Phoenix JDBC API's.
>>>>>
>>>>> In the above mention scenario, Are there any Phoenix functionalities (
>>>>> for ex: Salting, Secondary indexing) that will not be available because
>>>>> Phoenix JDBC driver is not used for inserting data.
>>>>>
>>>>> Can someone please share their thoughts on this?
>>>>>
>>>>> Hadoop Distro: CDH5
>>>>> HBase: 0.96.1
>>>>>
>>>>> --Unilocal
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> --Harshit
>
Re: Inserting data in Hbase Phoenix using the thrift api
Posted by Harshit Bapna <hr...@gmail.com>.
Hey Alex,
Thanks for creating the ticket.
I am currently discussing this internally and will need your help if we
decide to take this approach.
Another approach suggested by Jeffrey: Hack HBase thrift server to use
Phoenix instead of Hbase native api.
On Sun, May 11, 2014 at 1:40 AM, alex kamil <al...@gmail.com> wrote:
> I opened a ticket: https://issues.apache.org/jira/browse/PHOENIX-974 ,if
> it works out for you, contributions welcome)
> we use kafka as a buffer between app and db, it supports pretty high
> throughput <https://kafka.apache.org/07/performance.html> but may be not
> be suitable for very latency sensitive writes, your best bet is direct
> socket connection or some kind of distributed cache
>
>
>
> On Sun, May 11, 2014 at 4:06 AM, universal localhost <
> universal.localhost@gmail.com> wrote:
>
>> Thanks a lot James & Alex for your answer n suggestions.
>>
>> So *apart from Salting and Secondary Indexing, **are there any other
>> functionalities* *that I will loose by not using the phoenix JDBC api's
>> for inserting data*.
>> I think the above one is similar to 'Map to an existing HBase table'
>> case.
>>
>> The last option that u suggested rdg Hive ODBC driver sounds interesting
>> but I need to think a bit more on what it means :)
>> Rest all the options that you have suggested are nice but less suitable
>> for me as we aim to have a low lag between when the txn is first seen and
>> when its ready for analysis.
>>
>>
>> Alex, I didn't think about this approach. The C++ app server handles
>> close 50-60 tns per sec so I was a bit cautious but definitely worth a try.
>> Thnxs.
>>
>>
>> --Unilocal
>>
>>
>> On Wed, May 7, 2014 at 11:31 PM, James Taylor <ja...@apache.org>wrote:
>>
>>> Hi Unilocal,
>>> Yes, both salting and secondary indexing rely on the Phoenix client in
>>> cooperation with the server.
>>>
>>> Would it be possible for the C++ server to generate CSV files instead?
>>> Then these could be pumped into Phoenix through our CSV bulk loader (which
>>> could potentially be invoked through a variety of ways). Another
>>> alternative may be through our Apache Pig integration. Or it'd be pretty
>>> easy to adapt our Pig store func to a Hive SerDe. Then you could use the
>>> Hive ODBC driver to pump in data that's formated in a Phoenix compliant
>>> manner.
>>>
>>> If none of these are options, you could pump into a Phoenix table and
>>> then transfer the data (using Phoenix APIs) through UPSERT SELECT into a
>>> salted table or a table with secondary indexes.
>>>
>>> Thanks,
>>> James
>>>
>>>
>>> On Mon, May 5, 2014 at 2:42 PM, Localhost shell <
>>> universal.localhost@gmail.com> wrote:
>>>
>>>> Hey Folks,
>>>>
>>>> I have a use case where one of the apps(C++ server) will pump data into
>>>> the Hbase.
>>>> Since Phoenix doesn't support ODBC api's so the app will not be able to
>>>> use the Phoenix JDBC api and will use Hbase thirft api to insert the data.
>>>> Note: The app that is inserting data will create the row keys similar to
>>>> the way how Phoenix JDBC creates.
>>>>
>>>> Currently no data resides in Hbase and the table will be freshly
>>>> created using the SQL commands (using phoenix sqlline).
>>>> All the analysis/group-by queries will be triggered by a different app
>>>> using the Phoenix JDBC API's.
>>>>
>>>> In the above mention scenario, Are there any Phoenix functionalities (
>>>> for ex: Salting, Secondary indexing) that will not be available because
>>>> Phoenix JDBC driver is not used for inserting data.
>>>>
>>>> Can someone please share their thoughts on this?
>>>>
>>>> Hadoop Distro: CDH5
>>>> HBase: 0.96.1
>>>>
>>>> --Unilocal
>>>>
>>>>
>>>>
>>>
>>>
>>
>
--
--Harshit
Re: Inserting data in Hbase Phoenix using the thrift api
Posted by alex kamil <al...@gmail.com>.
I opened a ticket: https://issues.apache.org/jira/browse/PHOENIX-974 ,if
it works out for you, contributions welcome)
we use kafka as a buffer between app and db, it supports pretty high
throughput <https://kafka.apache.org/07/performance.html> but may be not be
suitable for very latency sensitive writes, your best bet is direct socket
connection or some kind of distributed cache
On Sun, May 11, 2014 at 4:06 AM, universal localhost <
universal.localhost@gmail.com> wrote:
> Thanks a lot James & Alex for your answer n suggestions.
>
> So *apart from Salting and Secondary Indexing, **are there any other
> functionalities* *that I will loose by not using the phoenix JDBC api's
> for inserting data*.
> I think the above one is similar to 'Map to an existing HBase table'
> case.
>
> The last option that u suggested rdg Hive ODBC driver sounds interesting
> but I need to think a bit more on what it means :)
> Rest all the options that you have suggested are nice but less suitable
> for me as we aim to have a low lag between when the txn is first seen and
> when its ready for analysis.
>
>
> Alex, I didn't think about this approach. The C++ app server handles close
> 50-60 tns per sec so I was a bit cautious but definitely worth a try.
> Thnxs.
>
>
> --Unilocal
>
>
> On Wed, May 7, 2014 at 11:31 PM, James Taylor <ja...@apache.org>wrote:
>
>> Hi Unilocal,
>> Yes, both salting and secondary indexing rely on the Phoenix client in
>> cooperation with the server.
>>
>> Would it be possible for the C++ server to generate CSV files instead?
>> Then these could be pumped into Phoenix through our CSV bulk loader (which
>> could potentially be invoked through a variety of ways). Another
>> alternative may be through our Apache Pig integration. Or it'd be pretty
>> easy to adapt our Pig store func to a Hive SerDe. Then you could use the
>> Hive ODBC driver to pump in data that's formated in a Phoenix compliant
>> manner.
>>
>> If none of these are options, you could pump into a Phoenix table and
>> then transfer the data (using Phoenix APIs) through UPSERT SELECT into a
>> salted table or a table with secondary indexes.
>>
>> Thanks,
>> James
>>
>>
>> On Mon, May 5, 2014 at 2:42 PM, Localhost shell <
>> universal.localhost@gmail.com> wrote:
>>
>>> Hey Folks,
>>>
>>> I have a use case where one of the apps(C++ server) will pump data into
>>> the Hbase.
>>> Since Phoenix doesn't support ODBC api's so the app will not be able to
>>> use the Phoenix JDBC api and will use Hbase thirft api to insert the data.
>>> Note: The app that is inserting data will create the row keys similar to
>>> the way how Phoenix JDBC creates.
>>>
>>> Currently no data resides in Hbase and the table will be freshly created
>>> using the SQL commands (using phoenix sqlline).
>>> All the analysis/group-by queries will be triggered by a different app
>>> using the Phoenix JDBC API's.
>>>
>>> In the above mention scenario, Are there any Phoenix functionalities (
>>> for ex: Salting, Secondary indexing) that will not be available because
>>> Phoenix JDBC driver is not used for inserting data.
>>>
>>> Can someone please share their thoughts on this?
>>>
>>> Hadoop Distro: CDH5
>>> HBase: 0.96.1
>>>
>>> --Unilocal
>>>
>>>
>>>
>>
>>
>
Re: Inserting data in Hbase Phoenix using the thrift api
Posted by universal localhost <un...@gmail.com>.
Thanks a lot James & Alex for your answer n suggestions.
So *apart from Salting and Secondary Indexing, **are there any other
functionalities* *that I will loose by not using the phoenix JDBC api's for
inserting data*.
I think the above one is similar to 'Map to an existing HBase table'
case.
The last option that u suggested rdg Hive ODBC driver sounds interesting
but I need to think a bit more on what it means :)
Rest all the options that you have suggested are nice but less suitable for
me as we aim to have a low lag between when the txn is first seen and when
its ready for analysis.
Alex, I didn't think about this approach. The C++ app server handles close
50-60 tns per sec so I was a bit cautious but definitely worth a try.
Thnxs.
--Unilocal
On Wed, May 7, 2014 at 11:31 PM, James Taylor <ja...@apache.org>wrote:
> Hi Unilocal,
> Yes, both salting and secondary indexing rely on the Phoenix client in
> cooperation with the server.
>
> Would it be possible for the C++ server to generate CSV files instead?
> Then these could be pumped into Phoenix through our CSV bulk loader (which
> could potentially be invoked through a variety of ways). Another
> alternative may be through our Apache Pig integration. Or it'd be pretty
> easy to adapt our Pig store func to a Hive SerDe. Then you could use the
> Hive ODBC driver to pump in data that's formated in a Phoenix compliant
> manner.
>
> If none of these are options, you could pump into a Phoenix table and then
> transfer the data (using Phoenix APIs) through UPSERT SELECT into a salted
> table or a table with secondary indexes.
>
> Thanks,
> James
>
>
> On Mon, May 5, 2014 at 2:42 PM, Localhost shell <
> universal.localhost@gmail.com> wrote:
>
>> Hey Folks,
>>
>> I have a use case where one of the apps(C++ server) will pump data into
>> the Hbase.
>> Since Phoenix doesn't support ODBC api's so the app will not be able to
>> use the Phoenix JDBC api and will use Hbase thirft api to insert the data.
>> Note: The app that is inserting data will create the row keys similar to
>> the way how Phoenix JDBC creates.
>>
>> Currently no data resides in Hbase and the table will be freshly created
>> using the SQL commands (using phoenix sqlline).
>> All the analysis/group-by queries will be triggered by a different app
>> using the Phoenix JDBC API's.
>>
>> In the above mention scenario, Are there any Phoenix functionalities (
>> for ex: Salting, Secondary indexing) that will not be available because
>> Phoenix JDBC driver is not used for inserting data.
>>
>> Can someone please share their thoughts on this?
>>
>> Hadoop Distro: CDH5
>> HBase: 0.96.1
>>
>> --Unilocal
>>
>>
>>
>
>
Re: Inserting data in Hbase Phoenix using the thrift api
Posted by James Taylor <ja...@apache.org>.
Hi Unilocal,
Yes, both salting and secondary indexing rely on the Phoenix client in
cooperation with the server.
Would it be possible for the C++ server to generate CSV files instead? Then
these could be pumped into Phoenix through our CSV bulk loader (which could
potentially be invoked through a variety of ways). Another alternative may
be through our Apache Pig integration. Or it'd be pretty easy to adapt our
Pig store func to a Hive SerDe. Then you could use the Hive ODBC driver to
pump in data that's formated in a Phoenix compliant manner.
If none of these are options, you could pump into a Phoenix table and then
transfer the data (using Phoenix APIs) through UPSERT SELECT into a salted
table or a table with secondary indexes.
Thanks,
James
On Mon, May 5, 2014 at 2:42 PM, Localhost shell <
universal.localhost@gmail.com> wrote:
> Hey Folks,
>
> I have a use case where one of the apps(C++ server) will pump data into
> the Hbase.
> Since Phoenix doesn't support ODBC api's so the app will not be able to
> use the Phoenix JDBC api and will use Hbase thirft api to insert the data.
> Note: The app that is inserting data will create the row keys similar to
> the way how Phoenix JDBC creates.
>
> Currently no data resides in Hbase and the table will be freshly created
> using the SQL commands (using phoenix sqlline).
> All the analysis/group-by queries will be triggered by a different app
> using the Phoenix JDBC API's.
>
> In the above mention scenario, Are there any Phoenix functionalities ( for
> ex: Salting, Secondary indexing) that will not be available because Phoenix
> JDBC driver is not used for inserting data.
>
> Can someone please share their thoughts on this?
>
> Hadoop Distro: CDH5
> HBase: 0.96.1
>
> --Unilocal
>
>
>