You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Billy Pearson <sa...@pearsonwholesale.com> on 2008/09/12 03:02:09 UTC
BatchUpdate and BatchOperation
should we not have a option to be able to get the BatchOperation's of a
BatchUpdate
Example
I have text files with lines of records I want to insert into hbase
I was just mapping them and the map was just inserting the records in to
hbase
I speed this up by adding inline code to detect multi columns per row that
are in order.
I was going to add a reduce stage to merge multi backupdates of the same row
in the
reduce stage but there is no way to get the BatchOperations from the
batchUpdates
Should we add a method to get the BatchOperations from the BatchUpdates?
Or Should I just be passing a BatchOperation to the reduce stage then merge
them into a BatchUpdate?
Either way I thank we need a method ("add" or something) for BatchUpdate
that accepts BatchOperations sense this is what we converting to in the end.
Re: BatchUpdate and BatchOperation
Posted by "Edward J. Yoon" <ed...@apache.org>.
Good +1, I agree with you.
-Ed
On Fri, Sep 12, 2008 at 10:02 AM, Billy Pearson
<sa...@pearsonwholesale.com> wrote:
> should we not have a option to be able to get the BatchOperation's of a
> BatchUpdate
>
> Example
>
> I have text files with lines of records I want to insert into hbase
> I was just mapping them and the map was just inserting the records in to
> hbase
>
> I speed this up by adding inline code to detect multi columns per row that
> are in order.
>
> I was going to add a reduce stage to merge multi backupdates of the same row
> in the
> reduce stage but there is no way to get the BatchOperations from the
> batchUpdates
>
> Should we add a method to get the BatchOperations from the BatchUpdates?
> Or Should I just be passing a BatchOperation to the reduce stage then merge
> them into a BatchUpdate?
>
> Either way I thank we need a method ("add" or something) for BatchUpdate
> that accepts BatchOperations sense this is what we converting to in the end.
>
>
>
--
Best regards, Edward J. Yoon
edwardyoon@apache.org
http://blog.udanax.org
Re: BatchUpdate and BatchOperation
Posted by Billy Pearson <bi...@sbcglobal.net>.
What I was doing was merging records for the same row on my map in a
BatchUpdate then was going to try and merge the BatchUpdates on the reduce
so all only inserting once per row.
But I have to pass BatchOperations on the map and build the BatchUpdate on
the Reduce sense we haven no way to get the column/value's of a BatchUpdate
or merge two BatchUpdates but I can see where it would be helpfull to be
able to get the column/values from a BatchUpdate.
Billy
----- Original Message -----
From: "Jim Kellerman" <ji...@public.gmane.org>
Newsgroups: gmane.comp.java.hadoop.hbase.user
To: <hb...@public.gmane.org>
Sent: Friday, September 12, 2008 11:09 AM
Subject: RE: BatchUpdate and BatchOperation
BatchUpdate implements Iterable<BatchOperation> doesn't that do what you
want?
---
Jim Kellerman, Senior Software Development Engineer
Powerset (Live Search, Microsoft Corporation)
> -----Original Message-----
> From: news [mailto:news-dbVV3NMTNubNLxjTenLetw@public.gmane.org] On Behalf
> Of Billy Pearson
> Sent: Thursday, September 11, 2008 6:02 PM
> To: hbase-user-7ArZoLwFLBtd/SJB6HiN2Ni2O/JbrIOy@public.gmane.org
> Subject: BatchUpdate and BatchOperation
>
> should we not have a option to be able to get the BatchOperation's of a
> BatchUpdate
>
> Example
>
> I have text files with lines of records I want to insert into hbase
> I was just mapping them and the map was just inserting the records in to
> hbase
>
> I speed this up by adding inline code to detect multi columns per row that
> are in order.
>
> I was going to add a reduce stage to merge multi backupdates of the same
> row
> in the
> reduce stage but there is no way to get the BatchOperations from the
> batchUpdates
>
> Should we add a method to get the BatchOperations from the BatchUpdates?
> Or Should I just be passing a BatchOperation to the reduce stage then
> merge
> them into a BatchUpdate?
>
> Either way I thank we need a method ("add" or something) for BatchUpdate
> that accepts BatchOperations sense this is what we converting to in the
> end.
>
Re: BatchUpdate and BatchOperation
Posted by Ryan Smith <ry...@gmail.com>.
I think hes saying theres no direct access to the list of batchOperations.
They are defined as private.
And i dont think there should be direct access. The operations should be
manipulated thru put/ get/ delete methods i think.
Just my 2 cents.
-Ryan
On Fri, Sep 12, 2008 at 12:09 PM, Jim Kellerman <ji...@powerset.com> wrote:
> BatchUpdate implements Iterable<BatchOperation> doesn't that do what you
> want?
>
> ---
> Jim Kellerman, Senior Software Development Engineer
> Powerset (Live Search, Microsoft Corporation)
>
>
> > -----Original Message-----
> > From: news [mailto:news@ger.gmane.org] On Behalf Of Billy Pearson
> > Sent: Thursday, September 11, 2008 6:02 PM
> > To: hbase-user@hadoop.apache.org
> > Subject: BatchUpdate and BatchOperation
> >
> > should we not have a option to be able to get the BatchOperation's of a
> > BatchUpdate
> >
> > Example
> >
> > I have text files with lines of records I want to insert into hbase
> > I was just mapping them and the map was just inserting the records in to
> > hbase
> >
> > I speed this up by adding inline code to detect multi columns per row
> that
> > are in order.
> >
> > I was going to add a reduce stage to merge multi backupdates of the same
> row
> > in the
> > reduce stage but there is no way to get the BatchOperations from the
> > batchUpdates
> >
> > Should we add a method to get the BatchOperations from the BatchUpdates?
> > Or Should I just be passing a BatchOperation to the reduce stage then
> merge
> > them into a BatchUpdate?
> >
> > Either way I thank we need a method ("add" or something) for BatchUpdate
> > that accepts BatchOperations sense this is what we converting to in the
> end.
> >
>
>
RE: BatchUpdate and BatchOperation
Posted by Jim Kellerman <ji...@powerset.com>.
BatchUpdate implements Iterable<BatchOperation> doesn't that do what you want?
---
Jim Kellerman, Senior Software Development Engineer
Powerset (Live Search, Microsoft Corporation)
> -----Original Message-----
> From: news [mailto:news@ger.gmane.org] On Behalf Of Billy Pearson
> Sent: Thursday, September 11, 2008 6:02 PM
> To: hbase-user@hadoop.apache.org
> Subject: BatchUpdate and BatchOperation
>
> should we not have a option to be able to get the BatchOperation's of a
> BatchUpdate
>
> Example
>
> I have text files with lines of records I want to insert into hbase
> I was just mapping them and the map was just inserting the records in to
> hbase
>
> I speed this up by adding inline code to detect multi columns per row that
> are in order.
>
> I was going to add a reduce stage to merge multi backupdates of the same row
> in the
> reduce stage but there is no way to get the BatchOperations from the
> batchUpdates
>
> Should we add a method to get the BatchOperations from the BatchUpdates?
> Or Should I just be passing a BatchOperation to the reduce stage then merge
> them into a BatchUpdate?
>
> Either way I thank we need a method ("add" or something) for BatchUpdate
> that accepts BatchOperations sense this is what we converting to in the end.
>
Re: BatchUpdate and BatchOperation
Posted by Billy Pearson <sa...@pearsonwholesale.com>.
Eather of the designs will be fine with me on HBASE-880
as for my post is just about adding a a new feature to BatchUpdate or what
ever it will be named in the future
Billy
"Jean-Daniel Cryans" <jd...@apache.org>
wrote in message
news:31a243e70809120647u7363cdcas14ffd53f0fa7b340@mail.gmail.com...
> Oh but it's not implemented yet, it is scheduled for 0.19.0. Please leave
> some comments in the jira regarding which design you prefer.
>
> Thx,
>
> J-D
>
> On Fri, Sep 12, 2008 at 1:37 AM, Billy Pearson
> <sa...@pearsonwholesale.com>wrote:
>
>> Thanks looks like HBASE-882 solves my problem in trunk
>> I am using 0.2.1 right now so I did not see it in the code.
>>
>> Billy
>>
>>
>> "Jean-Daniel Cryans" <jd...@apache.org>
>> wrote in message
>> news:31a243e70809111822n39493386m6bf798610ee48b3f@mail.gmail.com...
>>
>>> Billy,
>>>
>>> We are reworking the client API for 0.19.0. Have a look at
>>> HBASE-880<https://issues.apache.org/jira/browse/HBASE-880>and see if
>>>
>>> you can insert your methods in a proposed design!
>>>
>>> Thanks a lot,
>>>
>>> J-D
>>>
>>> On Thu, Sep 11, 2008 at 9:02 PM, Billy Pearson
>>> <sa...@pearsonwholesale.com>wrote:
>>>
>>> should we not have a option to be able to get the BatchOperation's of a
>>>> BatchUpdate
>>>>
>>>> Example
>>>>
>>>> I have text files with lines of records I want to insert into hbase
>>>> I was just mapping them and the map was just inserting the records in
>>>> to
>>>> hbase
>>>>
>>>> I speed this up by adding inline code to detect multi columns per row
>>>> that
>>>> are in order.
>>>>
>>>> I was going to add a reduce stage to merge multi backupdates of the
>>>> same
>>>> row in the
>>>> reduce stage but there is no way to get the BatchOperations from the
>>>> batchUpdates
>>>>
>>>> Should we add a method to get the BatchOperations from the
>>>> BatchUpdates?
>>>> Or Should I just be passing a BatchOperation to the reduce stage then
>>>> merge
>>>> them into a BatchUpdate?
>>>>
>>>> Either way I thank we need a method ("add" or something) for
>>>> BatchUpdate
>>>> that accepts BatchOperations sense this is what we converting to in the
>>>> end.
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: BatchUpdate and BatchOperation
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Oh but it's not implemented yet, it is scheduled for 0.19.0. Please leave
some comments in the jira regarding which design you prefer.
Thx,
J-D
On Fri, Sep 12, 2008 at 1:37 AM, Billy Pearson
<sa...@pearsonwholesale.com>wrote:
> Thanks looks like HBASE-882 solves my problem in trunk
> I am using 0.2.1 right now so I did not see it in the code.
>
> Billy
>
>
> "Jean-Daniel Cryans" <jd...@apache.org> wrote in message
> news:31a243e70809111822n39493386m6bf798610ee48b3f@mail.gmail.com...
>
>> Billy,
>>
>> We are reworking the client API for 0.19.0. Have a look at
>> HBASE-880<https://issues.apache.org/jira/browse/HBASE-880>and see if
>>
>> you can insert your methods in a proposed design!
>>
>> Thanks a lot,
>>
>> J-D
>>
>> On Thu, Sep 11, 2008 at 9:02 PM, Billy Pearson
>> <sa...@pearsonwholesale.com>wrote:
>>
>> should we not have a option to be able to get the BatchOperation's of a
>>> BatchUpdate
>>>
>>> Example
>>>
>>> I have text files with lines of records I want to insert into hbase
>>> I was just mapping them and the map was just inserting the records in to
>>> hbase
>>>
>>> I speed this up by adding inline code to detect multi columns per row
>>> that
>>> are in order.
>>>
>>> I was going to add a reduce stage to merge multi backupdates of the same
>>> row in the
>>> reduce stage but there is no way to get the BatchOperations from the
>>> batchUpdates
>>>
>>> Should we add a method to get the BatchOperations from the BatchUpdates?
>>> Or Should I just be passing a BatchOperation to the reduce stage then
>>> merge
>>> them into a BatchUpdate?
>>>
>>> Either way I thank we need a method ("add" or something) for BatchUpdate
>>> that accepts BatchOperations sense this is what we converting to in the
>>> end.
>>>
>>>
>>>
>>>
>>
>
>
Re: BatchUpdate and BatchOperation
Posted by Billy Pearson <sa...@pearsonwholesale.com>.
Thanks looks like HBASE-882 solves my problem in trunk
I am using 0.2.1 right now so I did not see it in the code.
Billy
"Jean-Daniel Cryans" <jd...@apache.org>
wrote in message
news:31a243e70809111822n39493386m6bf798610ee48b3f@mail.gmail.com...
> Billy,
>
> We are reworking the client API for 0.19.0. Have a look at
> HBASE-880<https://issues.apache.org/jira/browse/HBASE-880>and see if
> you can insert your methods in a proposed design!
>
> Thanks a lot,
>
> J-D
>
> On Thu, Sep 11, 2008 at 9:02 PM, Billy Pearson
> <sa...@pearsonwholesale.com>wrote:
>
>> should we not have a option to be able to get the BatchOperation's of a
>> BatchUpdate
>>
>> Example
>>
>> I have text files with lines of records I want to insert into hbase
>> I was just mapping them and the map was just inserting the records in to
>> hbase
>>
>> I speed this up by adding inline code to detect multi columns per row
>> that
>> are in order.
>>
>> I was going to add a reduce stage to merge multi backupdates of the same
>> row in the
>> reduce stage but there is no way to get the BatchOperations from the
>> batchUpdates
>>
>> Should we add a method to get the BatchOperations from the BatchUpdates?
>> Or Should I just be passing a BatchOperation to the reduce stage then
>> merge
>> them into a BatchUpdate?
>>
>> Either way I thank we need a method ("add" or something) for BatchUpdate
>> that accepts BatchOperations sense this is what we converting to in the
>> end.
>>
>>
>>
>
Re: BatchUpdate and BatchOperation
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Billy,
We are reworking the client API for 0.19.0. Have a look at
HBASE-880<https://issues.apache.org/jira/browse/HBASE-880>and see if
you can insert your methods in a proposed design!
Thanks a lot,
J-D
On Thu, Sep 11, 2008 at 9:02 PM, Billy Pearson
<sa...@pearsonwholesale.com>wrote:
> should we not have a option to be able to get the BatchOperation's of a
> BatchUpdate
>
> Example
>
> I have text files with lines of records I want to insert into hbase
> I was just mapping them and the map was just inserting the records in to
> hbase
>
> I speed this up by adding inline code to detect multi columns per row that
> are in order.
>
> I was going to add a reduce stage to merge multi backupdates of the same
> row in the
> reduce stage but there is no way to get the BatchOperations from the
> batchUpdates
>
> Should we add a method to get the BatchOperations from the BatchUpdates?
> Or Should I just be passing a BatchOperation to the reduce stage then merge
> them into a BatchUpdate?
>
> Either way I thank we need a method ("add" or something) for BatchUpdate
> that accepts BatchOperations sense this is what we converting to in the end.
>
>
>