You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Christopher Dorner <ch...@gmail.com> on 2011/11/02 18:14:42 UTC
Re: Creation of Hfiles for multiple tables using Single Bulk Load
Job?
Will HBase 0.92 support MultiHFileoutputFormat and IncrementalLoad for
different Tables?
Is there a comfortable way to make it work for HBase 0.90.4 as well? I
am using Cloudera's CDH3u2.
Am 30.10.2011 12:57, schrieb Christopher Dorner:
> Hi,
>
> i am facing a similar problem. I Need to read a large file to put into
> different hbase tables. Until now i have done it with
> MultiTableOutputFormat directly from the Mapper. Thats works ok, but i
> believe it will become quite slow when i try larger files. But I thought
> it is a good chance to try bulk load as well and compare them because i
> think it can be much faster in my scenario.
>
> It works fast and like a charm for one table. But i want to avoid
> reading the file a few times.
>
> Did you finally manage to make it work? Also with the lack of
> "configureIncrementalLoad" in the MultiHFileOutputFormat class? I am
> having only one family.
>
> Regards,
> Christopher
>
>
>
> Am -10.01.-28163 20:59, schrieb Arsalan Bilal:
>> Now I have redefine "MultiHfileOutputFormat" and "HfileOutputFormat"
>> code.
>> And it works fine for my solution. It generate data for multiple
>> families,
>> for multiple table and create multiple HFiles accordingly to related
>> Family
>> and save it at different paths.
>>
>> Thanks Andrew for indicating right direction and Thanks all community for
>> help
>>
>>
>>
>> On Tue, Sep 27, 2011 at 11:45 AM, Arsalan
>> Bilal<ch...@gmail.com>wrote:
>>
>>> Now this time, i am trying to achieve my objective by playing with
>>> HfileOutputFormat and MultiHfileOutputFormat. But did not achieve my
>>> target.
>>> So i need your help
>>> I am using two
>>> context.write(key, kv1);
>>> context.write(key, kv2);
>>> then it should write/generate data/Hfile on two different paths for two
>>> different tables. I am trying to achieve this. Plz help.
>>>
>>>
>>>
>>> On Mon, Sep 26, 2011 at 8:26 PM, Stack<st...@duboce.net> wrote:
>>>
>>>> Can you play around w/ the class to make it work for your case?
>>>> St.Ack
>>>>
>>>> On Mon, Sep 26, 2011 at 9:24 AM, Arsalan
>>>> Bilal<ch...@gmail.com>
>>>> wrote:
>>>>> @Andrew
>>>>>
>>>>> Can u help me? U suggest that Try this:
>>>>> https://gist.github.com/1237770, i
>>>>> saw it, How we can we use it actually?
>>>>> i try to use it as
>>>> job.setOutputFormatClass(MultiHFileOutputFormat.class);
>>>>> but it generate different column Families(not correct) inside one
>>>>> folder(recursively). I need to generate different column families in
>>>>> different folders.
>>>>> And there is also lack of configureIncrementalLoad Function in your
>>>> pointed
>>>>> file.
>>>>>
>>>>> Plz help me Andrew, Stack, i need your help.
>>>>>
>>>>>
>>>>> On Mon, Sep 26, 2011 at 6:48 PM, Shuja Rehman<sh...@gmail.com>
>>>> wrote:
>>>>>
>>>>>> @arslan,
>>>>>> I did not get chance to look at it.
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 26, 2011 at 12:11 PM, Arsalan Bilal<
>>>> charsalanbilal@gmail.com>wrote:
>>>>>>
>>>>>>> I am also facing same kind of problem, I am using one Map only and
>>>> need to
>>>>>>> generate multiple Hfiles for multiple table for insertion in Hbase?
>>>>>>> Have you found any solution ? Can u help me to find solution of this
>>>> one?
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 23, 2011 at 8:11 PM, Andrew Purtell<apurtell@apache.org
>>>>> wrote:
>>>>>>>
>>>>>>>> Try this: https://gist.github.com/1237770
>>>>>>>>
>>>>>>>> See line 135.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>>
>>>>>>>> - Andy
>>>>>>>>
>>>>>>>>
>>>>>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>>> Hein
>>>>>>>> (via Tom White)
>>>>>>>>
>>>>>>>>
>>>>>>>>> ________________________________
>>>>>>>>> From: Shuja Rehman<sh...@gmail.com>
>>>>>>>>> To: user@hbase.apache.org; Andrew Purtell<ap...@apache.org>
>>>>>>>>> Sent: Friday, September 23, 2011 8:18 AM
>>>>>>>>> Subject: Re: Creation of Hfiles for multiple tables using Single
>>>> Bulk
>>>>>>>> Load Job?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> andrew,
>>>>>>>>>
>>>>>>>>> can you guide me where to change and what to change? it will help.
>>>>>>>>> thnx
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Sep 22, 2011 at 9:35 PM, Shuja Rehman<
>>>> shujamughal@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Can you update it so I can use it? :)
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Sep 22, 2011 at 8:40 PM, Andrew Purtell<
>>>> apurtell@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> From: Shuja Rehman<sh...@gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I am using bulk load to insert data into hbase. Its runs
>>>>>>>>>>>> fine if
>>>> I
>>>>>>>> need to
>>>>>>>>>>>> insert in one table. But Now, I have the requirement in which I
>>>> need
>>>>>>>> to
>>>>>>>>>>>> insert data into more than one table.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> We started some support for this here:
>>>>>>>> https://issues.apache.org/jira/browse/HBASE-3727
>>>>>>>>>>>
>>>>>>>>>>> It needs to be updated for HFile v2, I think, a couple of single
>>>> line
>>>>>>>> changes.
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - Andy
>>>>>>>>>>>
>>>>>>>>>>> Problems worthy of attack prove their worth by hitting back. -
>>>> Piet
>>>>>>>> Hein (via Tom White)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>> From: Shuja Rehman<sh...@gmail.com>
>>>>>>>>>>>> To: user@hbase.apache.org
>>>>>>>>>>>> Cc:
>>>>>>>>>>>> Sent: Thursday, September 22, 2011 8:01 AM
>>>>>>>>>>>> Subject: Creation of Hfiles for multiple tables using Single
>>>> Bulk
>>>>>>>> Load Job?
>>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>> I am using bulk load to insert data into hbase. Its runs
>>>>>>>>>>>> fine if
>>>> I
>>>>>>>> need to
>>>>>>>>>>>> insert in one table. But Now, I have the requirement in which I
>>>> need
>>>>>>>> to
>>>>>>>>>>>> insert data into more than one table. But it seems that using
>>>> bulk
>>>>>>>> load, we
>>>>>>>>>>>> can generate hfiles for 1 table only. So the question is that
>>>> is
>>>>>>>> there any
>>>>>>>>>>>> solution which enable us to generate hfiles in multiple output
>>>> paths
>>>>>>>> and
>>>>>>>>>>>> then we can load these different files in different hbase
>>>> tables.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>>>
>>>>>>>>>>>> <http://pk.linkedin.com/in/shujamughal>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards
>>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards
>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Arsalan Bilal
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards
>>>>>> Shuja-ur-Rehman Baig
>>>>>> <http://pk.linkedin.com/in/shujamughal>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Arsalan Bilal
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Arsalan Bilal
>>>
>>>
>>>
>>
>>
>
Re: Creation of Hfiles for multiple tables using Single Bulk Load
Job?
Posted by Christopher Dorner <ch...@gmail.com>.
Thanks for your reply.
Yes, it would be great if he could publish his version.
Am 02.11.2011 21:27, schrieb Ted Yu:
> If Arsalan publishes the modified version of MultiHFileoutputFormat, that
> would expedite adding support in 0.94
> After some validation, this can be backported.
>
> Cheers
>
> On Wed, Nov 2, 2011 at 10:14 AM, Christopher Dorner<
> christopher.dorner@gmail.com> wrote:
>
>> Will HBase 0.92 support MultiHFileoutputFormat and IncrementalLoad for
>> different Tables?
>>
>> Is there a comfortable way to make it work for HBase 0.90.4 as well? I am
>> using Cloudera's CDH3u2.
>>
>>
>> Am 30.10.2011 12:57, schrieb Christopher Dorner:
>>
>> Hi,
>>>
>>> i am facing a similar problem. I Need to read a large file to put into
>>> different hbase tables. Until now i have done it with
>>> MultiTableOutputFormat directly from the Mapper. Thats works ok, but i
>>> believe it will become quite slow when i try larger files. But I thought
>>> it is a good chance to try bulk load as well and compare them because i
>>> think it can be much faster in my scenario.
>>>
>>> It works fast and like a charm for one table. But i want to avoid
>>> reading the file a few times.
>>>
>>> Did you finally manage to make it work? Also with the lack of
>>> "configureIncrementalLoad" in the MultiHFileOutputFormat class? I am
>>> having only one family.
>>>
>>> Regards,
>>> Christopher
>>>
>>>
>>>
>>> Am -10.01.-28163 20:59, schrieb Arsalan Bilal:
>>>
>>>> Now I have redefine "MultiHfileOutputFormat" and "HfileOutputFormat"
>>>> code.
>>>> And it works fine for my solution. It generate data for multiple
>>>> families,
>>>> for multiple table and create multiple HFiles accordingly to related
>>>> Family
>>>> and save it at different paths.
>>>>
>>>> Thanks Andrew for indicating right direction and Thanks all community for
>>>> help
>>>>
>>>>
>>>>
>>>> On Tue, Sep 27, 2011 at 11:45 AM, Arsalan
>>>> Bilal<charsalanbilal@gmail.com**>wrote:
>>>>
>>>> Now this time, i am trying to achieve my objective by playing with
>>>>> HfileOutputFormat and MultiHfileOutputFormat. But did not achieve my
>>>>> target.
>>>>> So i need your help
>>>>> I am using two
>>>>> context.write(key, kv1);
>>>>> context.write(key, kv2);
>>>>> then it should write/generate data/Hfile on two different paths for two
>>>>> different tables. I am trying to achieve this. Plz help.
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Sep 26, 2011 at 8:26 PM, Stack<st...@duboce.net> wrote:
>>>>>
>>>>> Can you play around w/ the class to make it work for your case?
>>>>>> St.Ack
>>>>>>
>>>>>> On Mon, Sep 26, 2011 at 9:24 AM, Arsalan
>>>>>> Bilal<charsalanbilal@gmail.com**>
>>>>>> wrote:
>>>>>>
>>>>>>> @Andrew
>>>>>>>
>>>>>>> Can u help me? U suggest that Try this:
>>>>>>> https://gist.github.com/**1237770<https://gist.github.com/1237770>,
>>>>>>> i
>>>>>>> saw it, How we can we use it actually?
>>>>>>> i try to use it as
>>>>>>>
>>>>>> job.setOutputFormatClass(**MultiHFileOutputFormat.class);
>>>>>>
>>>>>>> but it generate different column Families(not correct) inside one
>>>>>>> folder(recursively). I need to generate different column families in
>>>>>>> different folders.
>>>>>>> And there is also lack of configureIncrementalLoad Function in your
>>>>>>>
>>>>>> pointed
>>>>>>
>>>>>>> file.
>>>>>>>
>>>>>>> Plz help me Andrew, Stack, i need your help.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 26, 2011 at 6:48 PM, Shuja Rehman<sh...@gmail.com>
>>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> @arslan,
>>>>>>>> I did not get chance to look at it.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 26, 2011 at 12:11 PM, Arsalan Bilal<
>>>>>>>>
>>>>>>> charsalanbilal@gmail.com>**wrote:
>>>>>>
>>>>>>>
>>>>>>>> I am also facing same kind of problem, I am using one Map only and
>>>>>>>>>
>>>>>>>> need to
>>>>>>
>>>>>>> generate multiple Hfiles for multiple table for insertion in Hbase?
>>>>>>>>> Have you found any solution ? Can u help me to find solution of this
>>>>>>>>>
>>>>>>>> one?
>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Sep 23, 2011 at 8:11 PM, Andrew Purtell<apurtell@apache.org
>>>>>>>>>
>>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>> Try this: https://gist.github.com/**1237770<https://gist.github.com/1237770>
>>>>>>>>>>
>>>>>>>>>> See line 135.
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - Andy
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>>>>>>>>>
>>>>>>>>> Hein
>>>>>>
>>>>>>> (via Tom White)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ______________________________**__
>>>>>>>>>>> From: Shuja Rehman<sh...@gmail.com>
>>>>>>>>>>> To: user@hbase.apache.org; Andrew Purtell<ap...@apache.org>
>>>>>>>>>>> Sent: Friday, September 23, 2011 8:18 AM
>>>>>>>>>>> Subject: Re: Creation of Hfiles for multiple tables using Single
>>>>>>>>>>>
>>>>>>>>>> Bulk
>>>>>>
>>>>>>> Load Job?
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> andrew,
>>>>>>>>>>>
>>>>>>>>>>> can you guide me where to change and what to change? it will help.
>>>>>>>>>>> thnx
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Sep 22, 2011 at 9:35 PM, Shuja Rehman<
>>>>>>>>>>>
>>>>>>>>>> shujamughal@gmail.com>
>>>>>>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Can you update it so I can use it? :)
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Sep 22, 2011 at 8:40 PM, Andrew Purtell<
>>>>>>>>>>>>
>>>>>>>>>>> apurtell@apache.org>
>>>>>>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> From: Shuja Rehman<sh...@gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am using bulk load to insert data into hbase. Its runs
>>>>>>>>>>>>>> fine if
>>>>>>>>>>>>>>
>>>>>>>>>>>>> I
>>>>>>
>>>>>>> need to
>>>>>>>>>>
>>>>>>>>>>> insert in one table. But Now, I have the requirement in which I
>>>>>>>>>>>>>>
>>>>>>>>>>>>> need
>>>>>>
>>>>>>> to
>>>>>>>>>>
>>>>>>>>>>> insert data into more than one table.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> We started some support for this here:
>>>>>>>>>>>>>
>>>>>>>>>>>> https://issues.apache.org/**jira/browse/HBASE-3727<https://issues.apache.org/jira/browse/HBASE-3727>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> It needs to be updated for HFile v2, I think, a couple of single
>>>>>>>>>>>>>
>>>>>>>>>>>> line
>>>>>>
>>>>>>> changes.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Andy
>>>>>>>>>>>>>
>>>>>>>>>>>>> Problems worthy of attack prove their worth by hitting back. -
>>>>>>>>>>>>>
>>>>>>>>>>>> Piet
>>>>>>
>>>>>>> Hein (via Tom White)
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>
>>>>>>>>>>>>>> From: Shuja Rehman<sh...@gmail.com>
>>>>>>>>>>>>>> To: user@hbase.apache.org
>>>>>>>>>>>>>> Cc:
>>>>>>>>>>>>>> Sent: Thursday, September 22, 2011 8:01 AM
>>>>>>>>>>>>>> Subject: Creation of Hfiles for multiple tables using Single
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Bulk
>>>>>>
>>>>>>> Load Job?
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am using bulk load to insert data into hbase. Its runs
>>>>>>>>>>>>>> fine if
>>>>>>>>>>>>>>
>>>>>>>>>>>>> I
>>>>>>
>>>>>>> need to
>>>>>>>>>>
>>>>>>>>>>> insert in one table. But Now, I have the requirement in which I
>>>>>>>>>>>>>>
>>>>>>>>>>>>> need
>>>>>>
>>>>>>> to
>>>>>>>>>>
>>>>>>>>>>> insert data into more than one table. But it seems that using
>>>>>>>>>>>>>>
>>>>>>>>>>>>> bulk
>>>>>>
>>>>>>> load, we
>>>>>>>>>>
>>>>>>>>>>> can generate hfiles for 1 table only. So the question is that
>>>>>>>>>>>>>>
>>>>>>>>>>>>> is
>>>>>>
>>>>>>> there any
>>>>>>>>>>
>>>>>>>>>>> solution which enable us to generate hfiles in multiple output
>>>>>>>>>>>>>>
>>>>>>>>>>>>> paths
>>>>>>
>>>>>>> and
>>>>>>>>>>
>>>>>>>>>>> then we can load these different files in different hbase
>>>>>>>>>>>>>>
>>>>>>>>>>>>> tables.
>>>>>>
>>>>>>>
>>>>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> <http://pk.linkedin.com/in/**shujamughal<http://pk.linkedin.com/in/shujamughal>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards
>>>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards,
>>>>>>>>> Arsalan Bilal
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards
>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>> <http://pk.linkedin.com/in/**shujamughal<http://pk.linkedin.com/in/shujamughal>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Arsalan Bilal
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Arsalan Bilal
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Creation of Hfiles for multiple tables using Single Bulk Load Job?
Posted by Ted Yu <yu...@gmail.com>.
If Arsalan publishes the modified version of MultiHFileoutputFormat, that
would expedite adding support in 0.94
After some validation, this can be backported.
Cheers
On Wed, Nov 2, 2011 at 10:14 AM, Christopher Dorner <
christopher.dorner@gmail.com> wrote:
> Will HBase 0.92 support MultiHFileoutputFormat and IncrementalLoad for
> different Tables?
>
> Is there a comfortable way to make it work for HBase 0.90.4 as well? I am
> using Cloudera's CDH3u2.
>
>
> Am 30.10.2011 12:57, schrieb Christopher Dorner:
>
> Hi,
>>
>> i am facing a similar problem. I Need to read a large file to put into
>> different hbase tables. Until now i have done it with
>> MultiTableOutputFormat directly from the Mapper. Thats works ok, but i
>> believe it will become quite slow when i try larger files. But I thought
>> it is a good chance to try bulk load as well and compare them because i
>> think it can be much faster in my scenario.
>>
>> It works fast and like a charm for one table. But i want to avoid
>> reading the file a few times.
>>
>> Did you finally manage to make it work? Also with the lack of
>> "configureIncrementalLoad" in the MultiHFileOutputFormat class? I am
>> having only one family.
>>
>> Regards,
>> Christopher
>>
>>
>>
>> Am -10.01.-28163 20:59, schrieb Arsalan Bilal:
>>
>>> Now I have redefine "MultiHfileOutputFormat" and "HfileOutputFormat"
>>> code.
>>> And it works fine for my solution. It generate data for multiple
>>> families,
>>> for multiple table and create multiple HFiles accordingly to related
>>> Family
>>> and save it at different paths.
>>>
>>> Thanks Andrew for indicating right direction and Thanks all community for
>>> help
>>>
>>>
>>>
>>> On Tue, Sep 27, 2011 at 11:45 AM, Arsalan
>>> Bilal<charsalanbilal@gmail.com**>wrote:
>>>
>>> Now this time, i am trying to achieve my objective by playing with
>>>> HfileOutputFormat and MultiHfileOutputFormat. But did not achieve my
>>>> target.
>>>> So i need your help
>>>> I am using two
>>>> context.write(key, kv1);
>>>> context.write(key, kv2);
>>>> then it should write/generate data/Hfile on two different paths for two
>>>> different tables. I am trying to achieve this. Plz help.
>>>>
>>>>
>>>>
>>>> On Mon, Sep 26, 2011 at 8:26 PM, Stack<st...@duboce.net> wrote:
>>>>
>>>> Can you play around w/ the class to make it work for your case?
>>>>> St.Ack
>>>>>
>>>>> On Mon, Sep 26, 2011 at 9:24 AM, Arsalan
>>>>> Bilal<charsalanbilal@gmail.com**>
>>>>> wrote:
>>>>>
>>>>>> @Andrew
>>>>>>
>>>>>> Can u help me? U suggest that Try this:
>>>>>> https://gist.github.com/**1237770 <https://gist.github.com/1237770>,
>>>>>> i
>>>>>> saw it, How we can we use it actually?
>>>>>> i try to use it as
>>>>>>
>>>>> job.setOutputFormatClass(**MultiHFileOutputFormat.class);
>>>>>
>>>>>> but it generate different column Families(not correct) inside one
>>>>>> folder(recursively). I need to generate different column families in
>>>>>> different folders.
>>>>>> And there is also lack of configureIncrementalLoad Function in your
>>>>>>
>>>>> pointed
>>>>>
>>>>>> file.
>>>>>>
>>>>>> Plz help me Andrew, Stack, i need your help.
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 26, 2011 at 6:48 PM, Shuja Rehman<sh...@gmail.com>
>>>>>>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> @arslan,
>>>>>>> I did not get chance to look at it.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 26, 2011 at 12:11 PM, Arsalan Bilal<
>>>>>>>
>>>>>> charsalanbilal@gmail.com>**wrote:
>>>>>
>>>>>>
>>>>>>> I am also facing same kind of problem, I am using one Map only and
>>>>>>>>
>>>>>>> need to
>>>>>
>>>>>> generate multiple Hfiles for multiple table for insertion in Hbase?
>>>>>>>> Have you found any solution ? Can u help me to find solution of this
>>>>>>>>
>>>>>>> one?
>>>>>
>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Sep 23, 2011 at 8:11 PM, Andrew Purtell<apurtell@apache.org
>>>>>>>>
>>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>> Try this: https://gist.github.com/**1237770<https://gist.github.com/1237770>
>>>>>>>>>
>>>>>>>>> See line 135.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - Andy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>>>>>>>>
>>>>>>>> Hein
>>>>>
>>>>>> (via Tom White)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ______________________________**__
>>>>>>>>>> From: Shuja Rehman<sh...@gmail.com>
>>>>>>>>>> To: user@hbase.apache.org; Andrew Purtell<ap...@apache.org>
>>>>>>>>>> Sent: Friday, September 23, 2011 8:18 AM
>>>>>>>>>> Subject: Re: Creation of Hfiles for multiple tables using Single
>>>>>>>>>>
>>>>>>>>> Bulk
>>>>>
>>>>>> Load Job?
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> andrew,
>>>>>>>>>>
>>>>>>>>>> can you guide me where to change and what to change? it will help.
>>>>>>>>>> thnx
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Sep 22, 2011 at 9:35 PM, Shuja Rehman<
>>>>>>>>>>
>>>>>>>>> shujamughal@gmail.com>
>>>>>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Can you update it so I can use it? :)
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Sep 22, 2011 at 8:40 PM, Andrew Purtell<
>>>>>>>>>>>
>>>>>>>>>> apurtell@apache.org>
>>>>>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> From: Shuja Rehman<sh...@gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I am using bulk load to insert data into hbase. Its runs
>>>>>>>>>>>>> fine if
>>>>>>>>>>>>>
>>>>>>>>>>>> I
>>>>>
>>>>>> need to
>>>>>>>>>
>>>>>>>>>> insert in one table. But Now, I have the requirement in which I
>>>>>>>>>>>>>
>>>>>>>>>>>> need
>>>>>
>>>>>> to
>>>>>>>>>
>>>>>>>>>> insert data into more than one table.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> We started some support for this here:
>>>>>>>>>>>>
>>>>>>>>>>> https://issues.apache.org/**jira/browse/HBASE-3727<https://issues.apache.org/jira/browse/HBASE-3727>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> It needs to be updated for HFile v2, I think, a couple of single
>>>>>>>>>>>>
>>>>>>>>>>> line
>>>>>
>>>>>> changes.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> - Andy
>>>>>>>>>>>>
>>>>>>>>>>>> Problems worthy of attack prove their worth by hitting back. -
>>>>>>>>>>>>
>>>>>>>>>>> Piet
>>>>>
>>>>>> Hein (via Tom White)
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>
>>>>>>>>>>>>> From: Shuja Rehman<sh...@gmail.com>
>>>>>>>>>>>>> To: user@hbase.apache.org
>>>>>>>>>>>>> Cc:
>>>>>>>>>>>>> Sent: Thursday, September 22, 2011 8:01 AM
>>>>>>>>>>>>> Subject: Creation of Hfiles for multiple tables using Single
>>>>>>>>>>>>>
>>>>>>>>>>>> Bulk
>>>>>
>>>>>> Load Job?
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am using bulk load to insert data into hbase. Its runs
>>>>>>>>>>>>> fine if
>>>>>>>>>>>>>
>>>>>>>>>>>> I
>>>>>
>>>>>> need to
>>>>>>>>>
>>>>>>>>>> insert in one table. But Now, I have the requirement in which I
>>>>>>>>>>>>>
>>>>>>>>>>>> need
>>>>>
>>>>>> to
>>>>>>>>>
>>>>>>>>>> insert data into more than one table. But it seems that using
>>>>>>>>>>>>>
>>>>>>>>>>>> bulk
>>>>>
>>>>>> load, we
>>>>>>>>>
>>>>>>>>>> can generate hfiles for 1 table only. So the question is that
>>>>>>>>>>>>>
>>>>>>>>>>>> is
>>>>>
>>>>>> there any
>>>>>>>>>
>>>>>>>>>> solution which enable us to generate hfiles in multiple output
>>>>>>>>>>>>>
>>>>>>>>>>>> paths
>>>>>
>>>>>> and
>>>>>>>>>
>>>>>>>>>> then we can load these different files in different hbase
>>>>>>>>>>>>>
>>>>>>>>>>>> tables.
>>>>>
>>>>>>
>>>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <http://pk.linkedin.com/in/**shujamughal<http://pk.linkedin.com/in/shujamughal>
>>>>>>>>>>>>> >
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards
>>>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards
>>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>> Arsalan Bilal
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards
>>>>>>> Shuja-ur-Rehman Baig
>>>>>>> <http://pk.linkedin.com/in/**shujamughal<http://pk.linkedin.com/in/shujamughal>
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Arsalan Bilal
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Arsalan Bilal
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>