Unnecessary file copying during the bulkload: should we backport the fix in 0.96?

[pig user <pi...@gmail.com>]

In HBase 0.94, bulkload would always copy the files:

// Move the file if it's on another filesystem
FileSystem srcFs = srcPath.getFileSystem(conf);
if (!srcFs.equals(fs)) {
   LOG.info("File " + srcPath + " on different filesystem than " +
      "destination store - moving to this filesystem.");
    ......

Since fs here is an instance of HFileSystem. This would result in the
load taking long time to complete if the HFiles are in the destination
cluster.

This is fixed in trunk:

FileSystem srcFs = srcPath.getFileSystem(conf);
FileSystem desFs = fs instanceof HFileSystem ?
((HFileSystem)fs).getBackingFs() : fs;
    if (!srcFs.equals(desFs)) {
    ... ...

My question is: should we back port the fix to 0.94?

Re: Unnecessary file copying during the bulkload: should we backport the fix in 0.96?

[Stack <st...@duboce.net>]

On Thu, Sep 6, 2012 at 5:30 PM, pig user <pi...@gmail.com> wrote:
> In HBase 0.94, bulkload would always copy the files:
>
> // Move the file if it's on another filesystem
> FileSystem srcFs = srcPath.getFileSystem(conf);
> if (!srcFs.equals(fs)) {
>    LOG.info("File " + srcPath + " on different filesystem than " +
>       "destination store - moving to this filesystem.");
>     ......
>
> Since fs here is an instance of HFileSystem. This would result in the
> load taking long time to complete if the HFiles are in the destination
> cluster.
>
> This is fixed in trunk:
>
> FileSystem srcFs = srcPath.getFileSystem(conf);
> FileSystem desFs = fs instanceof HFileSystem ?
> ((HFileSystem)fs).getBackingFs() : fs;
>     if (!srcFs.equals(desFs)) {
>     ... ...
>
> My question is: should we back port the fix to 0.94?

Sure, make an issue and backport the patch and we'll commit it.
St.Ack

Re: Unnecessary file copying during the bulkload: should we backport the fix in 0.96?

[Richard Ding <pi...@gmail.com>]

Thanks Ted. It's great that the fix is already backported.

On Thu, Sep 6, 2012 at 7:19 PM, Ted Yu <yu...@gmail.com> wrote:
> The fix, HBASE-6529, would be in 0.94.2
>
> Expect 0.94.2 RC0 next week.
>
> Thanks
>
> On Thu, Sep 6, 2012 at 5:30 PM, pig user <pi...@gmail.com> wrote:
>
>> In HBase 0.94, bulkload would always copy the files:
>>
>> // Move the file if it's on another filesystem
>> FileSystem srcFs = srcPath.getFileSystem(conf);
>> if (!srcFs.equals(fs)) {
>>    LOG.info("File " + srcPath + " on different filesystem than " +
>>       "destination store - moving to this filesystem.");
>>     ......
>>
>> Since fs here is an instance of HFileSystem. This would result in the
>> load taking long time to complete if the HFiles are in the destination
>> cluster.
>>
>> This is fixed in trunk:
>>
>> FileSystem srcFs = srcPath.getFileSystem(conf);
>> FileSystem desFs = fs instanceof HFileSystem ?
>> ((HFileSystem)fs).getBackingFs() : fs;
>>     if (!srcFs.equals(desFs)) {
>>     ... ...
>>
>> My question is: should we back port the fix to 0.94?
>>

Re: Unnecessary file copying during the bulkload: should we backport the fix in 0.96?

[Ted Yu <yu...@gmail.com>]

The fix, HBASE-6529, would be in 0.94.2

Expect 0.94.2 RC0 next week.

Thanks

On Thu, Sep 6, 2012 at 5:30 PM, pig user <pi...@gmail.com> wrote:

> In HBase 0.94, bulkload would always copy the files:
>
> // Move the file if it's on another filesystem
> FileSystem srcFs = srcPath.getFileSystem(conf);
> if (!srcFs.equals(fs)) {
>    LOG.info("File " + srcPath + " on different filesystem than " +
>       "destination store - moving to this filesystem.");
>     ......
>
> Since fs here is an instance of HFileSystem. This would result in the
> load taking long time to complete if the HFiles are in the destination
> cluster.
>
> This is fixed in trunk:
>
> FileSystem srcFs = srcPath.getFileSystem(conf);
> FileSystem desFs = fs instanceof HFileSystem ?
> ((HFileSystem)fs).getBackingFs() : fs;
>     if (!srcFs.equals(desFs)) {
>     ... ...
>
> My question is: should we back port the fix to 0.94?
>