You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "M. BagherEsmaeily" <mb...@gmail.com> on 2013/09/14 08:31:29 UTC
completebulkload does 'copy' StoreFiles instead of 'cut'
Hello,
I was using HBase complete bulk load to transfer the output of ImportTsv to
a table in HBase, and I noticed that it copies the output instead of
cutting. This takes long time for my gigabytes of data.
In HBase documentation (
http://hbase.apache.org/book/ops_mgt.html#completebulkload) I read that the
files would be moved not copied. Can anyone help me with this?
Kind Regards
Re: completebulkload does 'copy' StoreFiles instead of 'cut'
Posted by "M. BagherEsmaeily" <mb...@gmail.com>.
When i use "hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles
hdfs://master.mydomain:8020/user/cluster/mbe/output MBE" instead of "hbase
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles mbe/output MBE",
the issue is resolved.
My problem was solved using absolute addressing with port number instead of
relative addressing.
I think that the difference between providing address with or without port
number is an issue and has to be resolved on next releases.
On Mon, Sep 16, 2013 at 10:20 AM, M. BagherEsmaeily <mb...@gmail.com>wrote:
> I noticed that the following line is in Region server log, which causes
> copying instead of cut:
>
> "on different filesystem than destination store - moving to this
> filesystem."
>
> This shows that source and destination store files are on different file
> systems, but both of them are on same HDFS. I just want to know what is the
> reason for this problem.
>
> Thanks
>
>
> On Sun, Sep 15, 2013 at 6:34 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> In the region server log file, do you see the following ?
>>
>> LOG.info("File " + srcPath + " on different filesystem than " +
>> "destination store - moving to this filesystem.");
>>
>>
>> On Sun, Sep 15, 2013 at 2:52 AM, M. BagherEsmaeily <mbesmaeily@gmail.com
>> >wrote:
>>
>> > Actually that was my fault duplicating the issue.I had no permission to
>> > delete the second issue. sorry!
>> >
>> >
>> > On Sun, Sep 15, 2013 at 2:12 PM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> > > Looks like HBASE-9538 is for the same issue.
>> > >
>> > > On Sep 15, 2013, at 1:09 AM, "M. BagherEsmaeily" <
>> mbesmaeily@gmail.com>
>> > > wrote:
>> > >
>> > > > HBASE-9537
>> > > >
>> > > >
>> > > > On Sun, Sep 15, 2013 at 11:56 AM, Ted Yu <yu...@gmail.com>
>> wrote:
>> > > >
>> > > >> File a JIRA for the issue ?
>> > > >>
>> > > >> On Sep 14, 2013, at 11:10 PM, "M. BagherEsmaeily" <
>> > mbesmaeily@gmail.com
>> > > >
>> > > >> wrote:
>> > > >>
>> > > >>> Hi rajesh,
>> > > >>>
>> > > >>> I use Hbase 0.94.11 and Hadoop 1.2.1. The file system of bulkload
>> > > >>> output directory and hbase cluster are the same, too.
>> > > >>>
>> > > >>> I've also coded a MapReduce job using HFileOutputFormat. When I
>> use
>> > > >>> LoadIncrementalHFiles
>> > > >>> to move the output of my job to HBase table, it still copies
>> instead
>> > of
>> > > >> cut.
>> > > >>>
>> > > >>> Thanks
>> > > >>>
>> > > >>>
>> > > >>> On Sat, Sep 14, 2013 at 2:50 PM, rajesh babu Chintaguntla <
>> > > >>> chrajeshbabu32@gmail.com> wrote:
>> > > >>>
>> > > >>>> Hi BagherEsmaeily,
>> > > >>>>
>> > > >>>> which version of hbase you are using? Is the file system of
>> > > >> bulkload
>> > > >>>> output directory and hbase cluster are same?
>> > > >>>>
>> > > >>>> If you are using hbase older than 0.94.5 version, the
>> > Storefiles
>> > > >>>> generated by importtsv are getting copied instead of
>> moving
>> > > even
>> > > >> if
>> > > >>>> the file system of bulkload output directory and hbase cluster
>> are
>> > > same.
>> > > >>>>
>> > > >>>> Its a bug and solved in 0.94.5 (HBASE-5498).
>> > > >>>>
>> > > >>>> Thanks.
>> > > >>>> Rajeshbabu
>> > > >>>>
>> > > >>>> On Sat, Sep 14, 2013 at 12:01 PM, M. BagherEsmaeily <
>> > > >> mbesmaeily@gmail.com
>> > > >>>>> wrote:
>> > > >>>>
>> > > >>>>> Hello,
>> > > >>>>>
>> > > >>>>> I was using HBase complete bulk load to transfer the output of
>> > > >> ImportTsv
>> > > >>>> to
>> > > >>>>> a table in HBase, and I noticed that it copies the output
>> instead
>> > of
>> > > >>>>> cutting. This takes long time for my gigabytes of data.
>> > > >>>>>
>> > > >>>>> In HBase documentation (
>> > > >>>>> http://hbase.apache.org/book/ops_mgt.html#completebulkload) I
>> read
>> > > >> that
>> > > >>>>> the
>> > > >>>>> files would be moved not copied. Can anyone help me with this?
>> > > >>>>>
>> > > >>>>> Kind Regards
>> > > >>
>> > >
>> >
>>
>
>
Re: completebulkload does 'copy' StoreFiles instead of 'cut'
Posted by "M. BagherEsmaeily" <mb...@gmail.com>.
I noticed that the following line is in Region server log, which causes
copying instead of cut:
"on different filesystem than destination store - moving to this
filesystem."
This shows that source and destination store files are on different file
systems, but both of them are on same HDFS. I just want to know what is the
reason for this problem.
Thanks
On Sun, Sep 15, 2013 at 6:34 PM, Ted Yu <yu...@gmail.com> wrote:
> In the region server log file, do you see the following ?
>
> LOG.info("File " + srcPath + " on different filesystem than " +
> "destination store - moving to this filesystem.");
>
>
> On Sun, Sep 15, 2013 at 2:52 AM, M. BagherEsmaeily <mbesmaeily@gmail.com
> >wrote:
>
> > Actually that was my fault duplicating the issue.I had no permission to
> > delete the second issue. sorry!
> >
> >
> > On Sun, Sep 15, 2013 at 2:12 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Looks like HBASE-9538 is for the same issue.
> > >
> > > On Sep 15, 2013, at 1:09 AM, "M. BagherEsmaeily" <mbesmaeily@gmail.com
> >
> > > wrote:
> > >
> > > > HBASE-9537
> > > >
> > > >
> > > > On Sun, Sep 15, 2013 at 11:56 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >
> > > >> File a JIRA for the issue ?
> > > >>
> > > >> On Sep 14, 2013, at 11:10 PM, "M. BagherEsmaeily" <
> > mbesmaeily@gmail.com
> > > >
> > > >> wrote:
> > > >>
> > > >>> Hi rajesh,
> > > >>>
> > > >>> I use Hbase 0.94.11 and Hadoop 1.2.1. The file system of bulkload
> > > >>> output directory and hbase cluster are the same, too.
> > > >>>
> > > >>> I've also coded a MapReduce job using HFileOutputFormat. When I use
> > > >>> LoadIncrementalHFiles
> > > >>> to move the output of my job to HBase table, it still copies
> instead
> > of
> > > >> cut.
> > > >>>
> > > >>> Thanks
> > > >>>
> > > >>>
> > > >>> On Sat, Sep 14, 2013 at 2:50 PM, rajesh babu Chintaguntla <
> > > >>> chrajeshbabu32@gmail.com> wrote:
> > > >>>
> > > >>>> Hi BagherEsmaeily,
> > > >>>>
> > > >>>> which version of hbase you are using? Is the file system of
> > > >> bulkload
> > > >>>> output directory and hbase cluster are same?
> > > >>>>
> > > >>>> If you are using hbase older than 0.94.5 version, the
> > Storefiles
> > > >>>> generated by importtsv are getting copied instead of moving
> > > even
> > > >> if
> > > >>>> the file system of bulkload output directory and hbase cluster are
> > > same.
> > > >>>>
> > > >>>> Its a bug and solved in 0.94.5 (HBASE-5498).
> > > >>>>
> > > >>>> Thanks.
> > > >>>> Rajeshbabu
> > > >>>>
> > > >>>> On Sat, Sep 14, 2013 at 12:01 PM, M. BagherEsmaeily <
> > > >> mbesmaeily@gmail.com
> > > >>>>> wrote:
> > > >>>>
> > > >>>>> Hello,
> > > >>>>>
> > > >>>>> I was using HBase complete bulk load to transfer the output of
> > > >> ImportTsv
> > > >>>> to
> > > >>>>> a table in HBase, and I noticed that it copies the output instead
> > of
> > > >>>>> cutting. This takes long time for my gigabytes of data.
> > > >>>>>
> > > >>>>> In HBase documentation (
> > > >>>>> http://hbase.apache.org/book/ops_mgt.html#completebulkload) I
> read
> > > >> that
> > > >>>>> the
> > > >>>>> files would be moved not copied. Can anyone help me with this?
> > > >>>>>
> > > >>>>> Kind Regards
> > > >>
> > >
> >
>
Re: completebulkload does 'copy' StoreFiles instead of 'cut'
Posted by lars hofhansl <la...@apache.org>.
This also happens when the split points of the HFiles to be imported do not match that of the table. In that case the files are split and the split files are written before they get moved into place.
________________________________
From: Ted Yu <yu...@gmail.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Sunday, September 15, 2013 7:04 AM
Subject: Re: completebulkload does 'copy' StoreFiles instead of 'cut'
In the region server log file, do you see the following ?
LOG.info("File " + srcPath + " on different filesystem than " +
"destination store - moving to this filesystem.");
On Sun, Sep 15, 2013 at 2:52 AM, M. BagherEsmaeily <mb...@gmail.com>wrote:
> Actually that was my fault duplicating the issue.I had no permission to
> delete the second issue. sorry!
>
>
> On Sun, Sep 15, 2013 at 2:12 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Looks like HBASE-9538 is for the same issue.
> >
> > On Sep 15, 2013, at 1:09 AM, "M. BagherEsmaeily" <mb...@gmail.com>
> > wrote:
> >
> > > HBASE-9537
> > >
> > >
> > > On Sun, Sep 15, 2013 at 11:56 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > >> File a JIRA for the issue ?
> > >>
> > >> On Sep 14, 2013, at 11:10 PM, "M. BagherEsmaeily" <
> mbesmaeily@gmail.com
> > >
> > >> wrote:
> > >>
> > >>> Hi rajesh,
> > >>>
> > >>> I use Hbase 0.94.11 and Hadoop 1.2.1. The file system of bulkload
> > >>> output directory and hbase cluster are the same, too.
> > >>>
> > >>> I've also coded a MapReduce job using HFileOutputFormat. When I use
> > >>> LoadIncrementalHFiles
> > >>> to move the output of my job to HBase table, it still copies instead
> of
> > >> cut.
> > >>>
> > >>> Thanks
> > >>>
> > >>>
> > >>> On Sat, Sep 14, 2013 at 2:50 PM, rajesh babu Chintaguntla <
> > >>> chrajeshbabu32@gmail.com> wrote:
> > >>>
> > >>>> Hi BagherEsmaeily,
> > >>>>
> > >>>> which version of hbase you are using? Is the file system of
> > >> bulkload
> > >>>> output directory and hbase cluster are same?
> > >>>>
> > >>>> If you are using hbase older than 0.94.5 version, the
> Storefiles
> > >>>> generated by importtsv are getting copied instead of moving
> > even
> > >> if
> > >>>> the file system of bulkload output directory and hbase cluster are
> > same.
> > >>>>
> > >>>> Its a bug and solved in 0.94.5 (HBASE-5498).
> > >>>>
> > >>>> Thanks.
> > >>>> Rajeshbabu
> > >>>>
> > >>>> On Sat, Sep 14, 2013 at 12:01 PM, M. BagherEsmaeily <
> > >> mbesmaeily@gmail.com
> > >>>>> wrote:
> > >>>>
> > >>>>> Hello,
> > >>>>>
> > >>>>> I was using HBase complete bulk load to transfer the output of
> > >> ImportTsv
> > >>>> to
> > >>>>> a table in HBase, and I noticed that it copies the output instead
> of
> > >>>>> cutting. This takes long time for my gigabytes of data.
> > >>>>>
> > >>>>> In HBase documentation (
> > >>>>> http://hbase.apache.org/book/ops_mgt.html#completebulkload) I read
> > >> that
> > >>>>> the
> > >>>>> files would be moved not copied. Can anyone help me with this?
> > >>>>>
> > >>>>> Kind Regards
> > >>
> >
>
Re: completebulkload does 'copy' StoreFiles instead of 'cut'
Posted by Ted Yu <yu...@gmail.com>.
In the region server log file, do you see the following ?
LOG.info("File " + srcPath + " on different filesystem than " +
"destination store - moving to this filesystem.");
On Sun, Sep 15, 2013 at 2:52 AM, M. BagherEsmaeily <mb...@gmail.com>wrote:
> Actually that was my fault duplicating the issue.I had no permission to
> delete the second issue. sorry!
>
>
> On Sun, Sep 15, 2013 at 2:12 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Looks like HBASE-9538 is for the same issue.
> >
> > On Sep 15, 2013, at 1:09 AM, "M. BagherEsmaeily" <mb...@gmail.com>
> > wrote:
> >
> > > HBASE-9537
> > >
> > >
> > > On Sun, Sep 15, 2013 at 11:56 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > >> File a JIRA for the issue ?
> > >>
> > >> On Sep 14, 2013, at 11:10 PM, "M. BagherEsmaeily" <
> mbesmaeily@gmail.com
> > >
> > >> wrote:
> > >>
> > >>> Hi rajesh,
> > >>>
> > >>> I use Hbase 0.94.11 and Hadoop 1.2.1. The file system of bulkload
> > >>> output directory and hbase cluster are the same, too.
> > >>>
> > >>> I've also coded a MapReduce job using HFileOutputFormat. When I use
> > >>> LoadIncrementalHFiles
> > >>> to move the output of my job to HBase table, it still copies instead
> of
> > >> cut.
> > >>>
> > >>> Thanks
> > >>>
> > >>>
> > >>> On Sat, Sep 14, 2013 at 2:50 PM, rajesh babu Chintaguntla <
> > >>> chrajeshbabu32@gmail.com> wrote:
> > >>>
> > >>>> Hi BagherEsmaeily,
> > >>>>
> > >>>> which version of hbase you are using? Is the file system of
> > >> bulkload
> > >>>> output directory and hbase cluster are same?
> > >>>>
> > >>>> If you are using hbase older than 0.94.5 version, the
> Storefiles
> > >>>> generated by importtsv are getting copied instead of moving
> > even
> > >> if
> > >>>> the file system of bulkload output directory and hbase cluster are
> > same.
> > >>>>
> > >>>> Its a bug and solved in 0.94.5 (HBASE-5498).
> > >>>>
> > >>>> Thanks.
> > >>>> Rajeshbabu
> > >>>>
> > >>>> On Sat, Sep 14, 2013 at 12:01 PM, M. BagherEsmaeily <
> > >> mbesmaeily@gmail.com
> > >>>>> wrote:
> > >>>>
> > >>>>> Hello,
> > >>>>>
> > >>>>> I was using HBase complete bulk load to transfer the output of
> > >> ImportTsv
> > >>>> to
> > >>>>> a table in HBase, and I noticed that it copies the output instead
> of
> > >>>>> cutting. This takes long time for my gigabytes of data.
> > >>>>>
> > >>>>> In HBase documentation (
> > >>>>> http://hbase.apache.org/book/ops_mgt.html#completebulkload) I read
> > >> that
> > >>>>> the
> > >>>>> files would be moved not copied. Can anyone help me with this?
> > >>>>>
> > >>>>> Kind Regards
> > >>
> >
>
Re: completebulkload does 'copy' StoreFiles instead of 'cut'
Posted by "M. BagherEsmaeily" <mb...@gmail.com>.
Actually that was my fault duplicating the issue.I had no permission to
delete the second issue. sorry!
On Sun, Sep 15, 2013 at 2:12 PM, Ted Yu <yu...@gmail.com> wrote:
> Looks like HBASE-9538 is for the same issue.
>
> On Sep 15, 2013, at 1:09 AM, "M. BagherEsmaeily" <mb...@gmail.com>
> wrote:
>
> > HBASE-9537
> >
> >
> > On Sun, Sep 15, 2013 at 11:56 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> File a JIRA for the issue ?
> >>
> >> On Sep 14, 2013, at 11:10 PM, "M. BagherEsmaeily" <mbesmaeily@gmail.com
> >
> >> wrote:
> >>
> >>> Hi rajesh,
> >>>
> >>> I use Hbase 0.94.11 and Hadoop 1.2.1. The file system of bulkload
> >>> output directory and hbase cluster are the same, too.
> >>>
> >>> I've also coded a MapReduce job using HFileOutputFormat. When I use
> >>> LoadIncrementalHFiles
> >>> to move the output of my job to HBase table, it still copies instead of
> >> cut.
> >>>
> >>> Thanks
> >>>
> >>>
> >>> On Sat, Sep 14, 2013 at 2:50 PM, rajesh babu Chintaguntla <
> >>> chrajeshbabu32@gmail.com> wrote:
> >>>
> >>>> Hi BagherEsmaeily,
> >>>>
> >>>> which version of hbase you are using? Is the file system of
> >> bulkload
> >>>> output directory and hbase cluster are same?
> >>>>
> >>>> If you are using hbase older than 0.94.5 version, the Storefiles
> >>>> generated by importtsv are getting copied instead of moving
> even
> >> if
> >>>> the file system of bulkload output directory and hbase cluster are
> same.
> >>>>
> >>>> Its a bug and solved in 0.94.5 (HBASE-5498).
> >>>>
> >>>> Thanks.
> >>>> Rajeshbabu
> >>>>
> >>>> On Sat, Sep 14, 2013 at 12:01 PM, M. BagherEsmaeily <
> >> mbesmaeily@gmail.com
> >>>>> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> I was using HBase complete bulk load to transfer the output of
> >> ImportTsv
> >>>> to
> >>>>> a table in HBase, and I noticed that it copies the output instead of
> >>>>> cutting. This takes long time for my gigabytes of data.
> >>>>>
> >>>>> In HBase documentation (
> >>>>> http://hbase.apache.org/book/ops_mgt.html#completebulkload) I read
> >> that
> >>>>> the
> >>>>> files would be moved not copied. Can anyone help me with this?
> >>>>>
> >>>>> Kind Regards
> >>
>
Re: completebulkload does 'copy' StoreFiles instead of 'cut'
Posted by Ted Yu <yu...@gmail.com>.
Looks like HBASE-9538 is for the same issue.
On Sep 15, 2013, at 1:09 AM, "M. BagherEsmaeily" <mb...@gmail.com> wrote:
> HBASE-9537
>
>
> On Sun, Sep 15, 2013 at 11:56 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> File a JIRA for the issue ?
>>
>> On Sep 14, 2013, at 11:10 PM, "M. BagherEsmaeily" <mb...@gmail.com>
>> wrote:
>>
>>> Hi rajesh,
>>>
>>> I use Hbase 0.94.11 and Hadoop 1.2.1. The file system of bulkload
>>> output directory and hbase cluster are the same, too.
>>>
>>> I've also coded a MapReduce job using HFileOutputFormat. When I use
>>> LoadIncrementalHFiles
>>> to move the output of my job to HBase table, it still copies instead of
>> cut.
>>>
>>> Thanks
>>>
>>>
>>> On Sat, Sep 14, 2013 at 2:50 PM, rajesh babu Chintaguntla <
>>> chrajeshbabu32@gmail.com> wrote:
>>>
>>>> Hi BagherEsmaeily,
>>>>
>>>> which version of hbase you are using? Is the file system of
>> bulkload
>>>> output directory and hbase cluster are same?
>>>>
>>>> If you are using hbase older than 0.94.5 version, the Storefiles
>>>> generated by importtsv are getting copied instead of moving even
>> if
>>>> the file system of bulkload output directory and hbase cluster are same.
>>>>
>>>> Its a bug and solved in 0.94.5 (HBASE-5498).
>>>>
>>>> Thanks.
>>>> Rajeshbabu
>>>>
>>>> On Sat, Sep 14, 2013 at 12:01 PM, M. BagherEsmaeily <
>> mbesmaeily@gmail.com
>>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I was using HBase complete bulk load to transfer the output of
>> ImportTsv
>>>> to
>>>>> a table in HBase, and I noticed that it copies the output instead of
>>>>> cutting. This takes long time for my gigabytes of data.
>>>>>
>>>>> In HBase documentation (
>>>>> http://hbase.apache.org/book/ops_mgt.html#completebulkload) I read
>> that
>>>>> the
>>>>> files would be moved not copied. Can anyone help me with this?
>>>>>
>>>>> Kind Regards
>>
Re: completebulkload does 'copy' StoreFiles instead of 'cut'
Posted by "M. BagherEsmaeily" <mb...@gmail.com>.
HBASE-9537
On Sun, Sep 15, 2013 at 11:56 AM, Ted Yu <yu...@gmail.com> wrote:
> File a JIRA for the issue ?
>
> On Sep 14, 2013, at 11:10 PM, "M. BagherEsmaeily" <mb...@gmail.com>
> wrote:
>
> > Hi rajesh,
> >
> > I use Hbase 0.94.11 and Hadoop 1.2.1. The file system of bulkload
> > output directory and hbase cluster are the same, too.
> >
> > I've also coded a MapReduce job using HFileOutputFormat. When I use
> > LoadIncrementalHFiles
> > to move the output of my job to HBase table, it still copies instead of
> cut.
> >
> > Thanks
> >
> >
> > On Sat, Sep 14, 2013 at 2:50 PM, rajesh babu Chintaguntla <
> > chrajeshbabu32@gmail.com> wrote:
> >
> >> Hi BagherEsmaeily,
> >>
> >> which version of hbase you are using? Is the file system of
> bulkload
> >> output directory and hbase cluster are same?
> >>
> >> If you are using hbase older than 0.94.5 version, the Storefiles
> >> generated by importtsv are getting copied instead of moving even
> if
> >> the file system of bulkload output directory and hbase cluster are same.
> >>
> >> Its a bug and solved in 0.94.5 (HBASE-5498).
> >>
> >> Thanks.
> >> Rajeshbabu
> >>
> >> On Sat, Sep 14, 2013 at 12:01 PM, M. BagherEsmaeily <
> mbesmaeily@gmail.com
> >>> wrote:
> >>
> >>> Hello,
> >>>
> >>> I was using HBase complete bulk load to transfer the output of
> ImportTsv
> >> to
> >>> a table in HBase, and I noticed that it copies the output instead of
> >>> cutting. This takes long time for my gigabytes of data.
> >>>
> >>> In HBase documentation (
> >>> http://hbase.apache.org/book/ops_mgt.html#completebulkload) I read
> that
> >>> the
> >>> files would be moved not copied. Can anyone help me with this?
> >>>
> >>> Kind Regards
> >>
>
Re: completebulkload does 'copy' StoreFiles instead of 'cut'
Posted by Ted Yu <yu...@gmail.com>.
File a JIRA for the issue ?
On Sep 14, 2013, at 11:10 PM, "M. BagherEsmaeily" <mb...@gmail.com> wrote:
> Hi rajesh,
>
> I use Hbase 0.94.11 and Hadoop 1.2.1. The file system of bulkload
> output directory and hbase cluster are the same, too.
>
> I've also coded a MapReduce job using HFileOutputFormat. When I use
> LoadIncrementalHFiles
> to move the output of my job to HBase table, it still copies instead of cut.
>
> Thanks
>
>
> On Sat, Sep 14, 2013 at 2:50 PM, rajesh babu Chintaguntla <
> chrajeshbabu32@gmail.com> wrote:
>
>> Hi BagherEsmaeily,
>>
>> which version of hbase you are using? Is the file system of bulkload
>> output directory and hbase cluster are same?
>>
>> If you are using hbase older than 0.94.5 version, the Storefiles
>> generated by importtsv are getting copied instead of moving even if
>> the file system of bulkload output directory and hbase cluster are same.
>>
>> Its a bug and solved in 0.94.5 (HBASE-5498).
>>
>> Thanks.
>> Rajeshbabu
>>
>> On Sat, Sep 14, 2013 at 12:01 PM, M. BagherEsmaeily <mbesmaeily@gmail.com
>>> wrote:
>>
>>> Hello,
>>>
>>> I was using HBase complete bulk load to transfer the output of ImportTsv
>> to
>>> a table in HBase, and I noticed that it copies the output instead of
>>> cutting. This takes long time for my gigabytes of data.
>>>
>>> In HBase documentation (
>>> http://hbase.apache.org/book/ops_mgt.html#completebulkload) I read that
>>> the
>>> files would be moved not copied. Can anyone help me with this?
>>>
>>> Kind Regards
>>
Re: completebulkload does 'copy' StoreFiles instead of 'cut'
Posted by "M. BagherEsmaeily" <mb...@gmail.com>.
Hi rajesh,
I use Hbase 0.94.11 and Hadoop 1.2.1. The file system of bulkload
output directory and hbase cluster are the same, too.
I've also coded a MapReduce job using HFileOutputFormat. When I use
LoadIncrementalHFiles
to move the output of my job to HBase table, it still copies instead of cut.
Thanks
On Sat, Sep 14, 2013 at 2:50 PM, rajesh babu Chintaguntla <
chrajeshbabu32@gmail.com> wrote:
> Hi BagherEsmaeily,
>
> which version of hbase you are using? Is the file system of bulkload
> output directory and hbase cluster are same?
>
> If you are using hbase older than 0.94.5 version, the Storefiles
> generated by importtsv are getting copied instead of moving even if
> the file system of bulkload output directory and hbase cluster are same.
>
> Its a bug and solved in 0.94.5 (HBASE-5498).
>
> Thanks.
> Rajeshbabu
>
> On Sat, Sep 14, 2013 at 12:01 PM, M. BagherEsmaeily <mbesmaeily@gmail.com
> >wrote:
>
> > Hello,
> >
> > I was using HBase complete bulk load to transfer the output of ImportTsv
> to
> > a table in HBase, and I noticed that it copies the output instead of
> > cutting. This takes long time for my gigabytes of data.
> >
> > In HBase documentation (
> > http://hbase.apache.org/book/ops_mgt.html#completebulkload) I read that
> > the
> > files would be moved not copied. Can anyone help me with this?
> >
> > Kind Regards
> >
>
Re: completebulkload does 'copy' StoreFiles instead of 'cut'
Posted by rajesh babu Chintaguntla <ch...@gmail.com>.
Hi BagherEsmaeily,
which version of hbase you are using? Is the file system of bulkload
output directory and hbase cluster are same?
If you are using hbase older than 0.94.5 version, the Storefiles
generated by importtsv are getting copied instead of moving even if
the file system of bulkload output directory and hbase cluster are same.
Its a bug and solved in 0.94.5 (HBASE-5498).
Thanks.
Rajeshbabu
On Sat, Sep 14, 2013 at 12:01 PM, M. BagherEsmaeily <mb...@gmail.com>wrote:
> Hello,
>
> I was using HBase complete bulk load to transfer the output of ImportTsv to
> a table in HBase, and I noticed that it copies the output instead of
> cutting. This takes long time for my gigabytes of data.
>
> In HBase documentation (
> http://hbase.apache.org/book/ops_mgt.html#completebulkload) I read that
> the
> files would be moved not copied. Can anyone help me with this?
>
> Kind Regards
>