You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Austin Heyne <ah...@ccri.com> on 2018/06/20 23:07:53 UTC

Bulk Load is pulling files locally with same FS and validation disabled.

Hi everyone,

I'm trying to run a bulk load of about 15TB of data sitting in s3 that 
I've bulk ingested. When I initiate the load I'm seeing the data get 
copied down to the workers and then back up to s3 even though the HBase 
root is that same bucket.

Data to be bulk loaded is at s3://bucket_name/data/bulkload/z3/d/ and 
the HBase root is s3://bucket_name/. I have validation disabled through 
the "hbase.loadincremental.validate.hfile" = "false" config set in code 
before I call LoadIncrementalHFiles.doBulkLoad. (Code is available at 
[1]) The splits have already been generated with the same config that 
was used during the ingest so they'll line up. I'm currently running 
HBase 1.4.2 on AWS EMR. The logs of interest were pulled from a worker 
on the cluster:

"""
2018-06-20 22:42:15,888 INFO 
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
s3n.S3NativeFileSystem: Opening 
's3n://bucket_name/data/bulkload/z3/d/b9371885084e4060ac157799e5c89b59' 
for reading
2018-06-20 22:42:16,026 INFO 
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
compress.CodecPool: Got brand-new decompressor [.snappy]
2018-06-20 22:42:16,056 INFO 
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
regionserver.HRegionFileSystem: Bulk-load file 
s3n://bucket_name/data/bulkload/z3/d/b9371885084e4060ac157799e5c89b59 is 
on different filesystem than the destination store. Copying file over to 
destination filesystem.
2018-06-20 22:42:16,109 INFO 
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
s3n.S3NativeFileSystem: Opening 
's3n://bucket_name/data/bulkload/z3/d/b9371885084e4060ac157799e5c89b59' 
for reading
2018-06-20 22:42:17,910 INFO 
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
s3n.MultipartUploadOutputStream: close closed:false 
s3://bucket_name/data/default/rtb_rtb_z3_v2/c34a364639dab40857a8d18871a3f705/.tmp/3d69598daa9841f986da2341a5901444
2018-06-20 22:42:17,927 INFO 
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
regionserver.HRegionFileSystem: Copied 
s3n://bucket_name/data/bulkload/z3/d/b9371885084e4060ac157799e5c89b59 to 
temporary path on destination filesystem: 
s3://bucket_name/data/default/rtb_rtb_z3_v2/c34a364639dab40857a8d18871a3f705/.tmp/3d69598daa9841f986da2341a5901444
"""

I'm seeing the behavior using 's3://' or 's3n://'. Has anyone 
experienced this or have advice?

Thanks,
Austin Heyne

[1] 
https://github.com/aheyne/geomesa/blob/b36507f4e999b295ebab8b1fb47a38f53e2a0e93/geomesa-hbase/geomesa-hbase-tools/src/main/scala/org/locationtech/geomesa/hbase/tools/ingest/HBaseBulkLoadCommand.scala#L47 


Re: Bulk Load is pulling files locally with same FS and validation disabled.

Posted by Austin Heyne <ah...@ccri.com>.
Sure thing, I've created a ticket here [1]. Though since we are using 
EMR it's not simple to swap out this code so I would still be very 
interested in a workaround if anyone has an idea.

Thanks,
Austin Heyne

[1] https://issues.apache.org/jira/browse/HBASE-20774


On 06/21/2018 06:48 PM, Ted Yu wrote:
> Since S3FileSystem is not taken into account in FSHDFSUtils#isSameHdfs, we
> need to add more code to avoid the overhead.
>
> Can you log a JIRA with what you discovered ?
>
> Thanks
>
> On Thu, Jun 21, 2018 at 2:08 PM, Austin Heyne <ah...@ccri.com> wrote:
>
>> Hi again,
>>
>> I've been doing more digging into this and I've found that with the way
>> the code it written it's actually impossible. In FSHDFSUtils [1] HBase
>> attempts to get the Canonical Service Name from Hadoop. Since we're running
>> on EMR our filesystem is the S3NativeFileSystem (com.amazon) which extends,
>> I believe, the NativeS3FileSystem (org.apache). Since the
>> NativeS3FileSystem [2] and S3FileSystem [2] both always return null for
>> getCannonicalServiceName and from testing it appears the S3NativeFileSystem
>> does the same it looks like there is no way to get past the check in
>> FSHDFSUtils.isSameHdfs when running on S3 of any kind.
>>
>> Does anyone know of a workaround for this issue?
>>
>> Thanks,
>> Austin Heyne
>>
>> [1] https://github.com/apache/hbase/blob/rel/1.4.2/hbase-server/
>> src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java#L120-L123
>> [2] https://github.com/apache/hadoop/blob/release-2.8.3-RC0/hado
>> op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/
>> s3native/NativeS3FileSystem.java#L788-L791
>> [3] https://github.com/apache/hadoop/blob/release-2.8.3-RC0/hado
>> op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3/
>> S3FileSystem.java#L403-L406
>>
>>
>> On 06/20/2018 07:07 PM, Austin Heyne wrote:
>>
>>> Hi everyone,
>>>
>>> I'm trying to run a bulk load of about 15TB of data sitting in s3 that
>>> I've bulk ingested. When I initiate the load I'm seeing the data get copied
>>> down to the workers and then back up to s3 even though the HBase root is
>>> that same bucket.
>>>
>>> Data to be bulk loaded is at s3://bucket_name/data/bulkload/z3/d/ and
>>> the HBase root is s3://bucket_name/. I have validation disabled through the
>>> "hbase.loadincremental.validate.hfile" = "false" config set in code
>>> before I call LoadIncrementalHFiles.doBulkLoad. (Code is available at
>>> [1]) The splits have already been generated with the same config that was
>>> used during the ingest so they'll line up. I'm currently running HBase
>>> 1.4.2 on AWS EMR. The logs of interest were pulled from a worker on the
>>> cluster:
>>>
>>> """
>>> 2018-06-20 22:42:15,888 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>>> s3n.S3NativeFileSystem: Opening 's3n://bucket_name/data/bulklo
>>> ad/z3/d/b9371885084e4060ac157799e5c89b59' for reading
>>> 2018-06-20 22:42:16,026 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>>> compress.CodecPool: Got brand-new decompressor [.snappy]
>>> 2018-06-20 22:42:16,056 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>>> regionserver.HRegionFileSystem: Bulk-load file
>>> s3n://bucket_name/data/bulkload/z3/d/b9371885084e4060ac157799e5c89b59 is
>>> on different filesystem than the destination store. Copying file over to
>>> destination filesystem.
>>> 2018-06-20 22:42:16,109 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>>> s3n.S3NativeFileSystem: Opening 's3n://bucket_name/data/bulklo
>>> ad/z3/d/b9371885084e4060ac157799e5c89b59' for reading
>>> 2018-06-20 22:42:17,910 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>>> s3n.MultipartUploadOutputStream: close closed:false
>>> s3://bucket_name/data/default/z3_v2/c34a364639dab40857a8d188
>>> 71a3f705/.tmp/3d69598daa9841f986da2341a5901444
>>> 2018-06-20 22:42:17,927 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>>> regionserver.HRegionFileSystem: Copied s3n://bucket_name/data/bulkloa
>>> d/z3/d/b9371885084e4060ac157799e5c89b59 to temporary path on destination
>>> filesystem: s3://bucket_name/data/default/z3_v2/c34a364639dab40857a8d188
>>> 71a3f705/.tmp/3d69598daa9841f986da2341a5901444
>>> """
>>>
>>> I'm seeing the behavior using 's3://' or 's3n://'. Has anyone experienced
>>> this or have advice?
>>>
>>> Thanks,
>>> Austin Heyne
>>>
>>> [1] https://github.com/aheyne/geomesa/blob/b36507f4e999b295ebab8
>>> b1fb47a38f53e2a0e93/geomesa-hbase/geomesa-hbase-tools/src/
>>> main/scala/org/locationtech/geomesa/hbase/tools/ingest/HBase
>>> BulkLoadCommand.scala#L47
>>>
>> --
>> Austin L. Heyne
>>
>>

-- 
Austin L. Heyne


Re: Bulk Load is pulling files locally with same FS and validation disabled.

Posted by Ted Yu <yu...@gmail.com>.
Since S3FileSystem is not taken into account in FSHDFSUtils#isSameHdfs, we
need to add more code to avoid the overhead.

Can you log a JIRA with what you discovered ?

Thanks

On Thu, Jun 21, 2018 at 2:08 PM, Austin Heyne <ah...@ccri.com> wrote:

> Hi again,
>
> I've been doing more digging into this and I've found that with the way
> the code it written it's actually impossible. In FSHDFSUtils [1] HBase
> attempts to get the Canonical Service Name from Hadoop. Since we're running
> on EMR our filesystem is the S3NativeFileSystem (com.amazon) which extends,
> I believe, the NativeS3FileSystem (org.apache). Since the
> NativeS3FileSystem [2] and S3FileSystem [2] both always return null for
> getCannonicalServiceName and from testing it appears the S3NativeFileSystem
> does the same it looks like there is no way to get past the check in
> FSHDFSUtils.isSameHdfs when running on S3 of any kind.
>
> Does anyone know of a workaround for this issue?
>
> Thanks,
> Austin Heyne
>
> [1] https://github.com/apache/hbase/blob/rel/1.4.2/hbase-server/
> src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java#L120-L123
> [2] https://github.com/apache/hadoop/blob/release-2.8.3-RC0/hado
> op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/
> s3native/NativeS3FileSystem.java#L788-L791
> [3] https://github.com/apache/hadoop/blob/release-2.8.3-RC0/hado
> op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3/
> S3FileSystem.java#L403-L406
>
>
> On 06/20/2018 07:07 PM, Austin Heyne wrote:
>
>> Hi everyone,
>>
>> I'm trying to run a bulk load of about 15TB of data sitting in s3 that
>> I've bulk ingested. When I initiate the load I'm seeing the data get copied
>> down to the workers and then back up to s3 even though the HBase root is
>> that same bucket.
>>
>> Data to be bulk loaded is at s3://bucket_name/data/bulkload/z3/d/ and
>> the HBase root is s3://bucket_name/. I have validation disabled through the
>> "hbase.loadincremental.validate.hfile" = "false" config set in code
>> before I call LoadIncrementalHFiles.doBulkLoad. (Code is available at
>> [1]) The splits have already been generated with the same config that was
>> used during the ingest so they'll line up. I'm currently running HBase
>> 1.4.2 on AWS EMR. The logs of interest were pulled from a worker on the
>> cluster:
>>
>> """
>> 2018-06-20 22:42:15,888 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>> s3n.S3NativeFileSystem: Opening 's3n://bucket_name/data/bulklo
>> ad/z3/d/b9371885084e4060ac157799e5c89b59' for reading
>> 2018-06-20 22:42:16,026 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>> compress.CodecPool: Got brand-new decompressor [.snappy]
>> 2018-06-20 22:42:16,056 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>> regionserver.HRegionFileSystem: Bulk-load file
>> s3n://bucket_name/data/bulkload/z3/d/b9371885084e4060ac157799e5c89b59 is
>> on different filesystem than the destination store. Copying file over to
>> destination filesystem.
>> 2018-06-20 22:42:16,109 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>> s3n.S3NativeFileSystem: Opening 's3n://bucket_name/data/bulklo
>> ad/z3/d/b9371885084e4060ac157799e5c89b59' for reading
>> 2018-06-20 22:42:17,910 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>> s3n.MultipartUploadOutputStream: close closed:false
>> s3://bucket_name/data/default/z3_v2/c34a364639dab40857a8d188
>> 71a3f705/.tmp/3d69598daa9841f986da2341a5901444
>> 2018-06-20 22:42:17,927 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>> regionserver.HRegionFileSystem: Copied s3n://bucket_name/data/bulkloa
>> d/z3/d/b9371885084e4060ac157799e5c89b59 to temporary path on destination
>> filesystem: s3://bucket_name/data/default/z3_v2/c34a364639dab40857a8d188
>> 71a3f705/.tmp/3d69598daa9841f986da2341a5901444
>> """
>>
>> I'm seeing the behavior using 's3://' or 's3n://'. Has anyone experienced
>> this or have advice?
>>
>> Thanks,
>> Austin Heyne
>>
>> [1] https://github.com/aheyne/geomesa/blob/b36507f4e999b295ebab8
>> b1fb47a38f53e2a0e93/geomesa-hbase/geomesa-hbase-tools/src/
>> main/scala/org/locationtech/geomesa/hbase/tools/ingest/HBase
>> BulkLoadCommand.scala#L47
>>
>
> --
> Austin L. Heyne
>
>

Re: Bulk Load is pulling files locally with same FS and validation disabled.

Posted by Austin Heyne <ah...@ccri.com>.
Hi again,

I've been doing more digging into this and I've found that with the way 
the code it written it's actually impossible. In FSHDFSUtils [1] HBase 
attempts to get the Canonical Service Name from Hadoop. Since we're 
running on EMR our filesystem is the S3NativeFileSystem (com.amazon) 
which extends, I believe, the NativeS3FileSystem (org.apache). Since the 
NativeS3FileSystem [2] and S3FileSystem [2] both always return null for 
getCannonicalServiceName and from testing it appears the 
S3NativeFileSystem does the same it looks like there is no way to get 
past the check in FSHDFSUtils.isSameHdfs when running on S3 of any kind.

Does anyone know of a workaround for this issue?

Thanks,
Austin Heyne

[1] 
https://github.com/apache/hbase/blob/rel/1.4.2/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java#L120-L123
[2] 
https://github.com/apache/hadoop/blob/release-2.8.3-RC0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java#L788-L791
[3] 
https://github.com/apache/hadoop/blob/release-2.8.3-RC0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java#L403-L406


On 06/20/2018 07:07 PM, Austin Heyne wrote:
> Hi everyone,
>
> I'm trying to run a bulk load of about 15TB of data sitting in s3 that 
> I've bulk ingested. When I initiate the load I'm seeing the data get 
> copied down to the workers and then back up to s3 even though the 
> HBase root is that same bucket.
>
> Data to be bulk loaded is at s3://bucket_name/data/bulkload/z3/d/ and 
> the HBase root is s3://bucket_name/. I have validation disabled 
> through the "hbase.loadincremental.validate.hfile" = "false" config 
> set in code before I call LoadIncrementalHFiles.doBulkLoad. (Code is 
> available at [1]) The splits have already been generated with the same 
> config that was used during the ingest so they'll line up. I'm 
> currently running HBase 1.4.2 on AWS EMR. The logs of interest were 
> pulled from a worker on the cluster:
>
> """
> 2018-06-20 22:42:15,888 INFO 
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> s3n.S3NativeFileSystem: Opening 
> 's3n://bucket_name/data/bulkload/z3/d/b9371885084e4060ac157799e5c89b59' 
> for reading
> 2018-06-20 22:42:16,026 INFO 
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> compress.CodecPool: Got brand-new decompressor [.snappy]
> 2018-06-20 22:42:16,056 INFO 
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> regionserver.HRegionFileSystem: Bulk-load file 
> s3n://bucket_name/data/bulkload/z3/d/b9371885084e4060ac157799e5c89b59 
> is on different filesystem than the destination store. Copying file 
> over to destination filesystem.
> 2018-06-20 22:42:16,109 INFO 
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> s3n.S3NativeFileSystem: Opening 
> 's3n://bucket_name/data/bulkload/z3/d/b9371885084e4060ac157799e5c89b59' 
> for reading
> 2018-06-20 22:42:17,910 INFO 
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> s3n.MultipartUploadOutputStream: close closed:false 
> s3://bucket_name/data/default/z3_v2/c34a364639dab40857a8d18871a3f705/.tmp/3d69598daa9841f986da2341a5901444
> 2018-06-20 22:42:17,927 INFO 
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> regionserver.HRegionFileSystem: Copied 
> s3n://bucket_name/data/bulkload/z3/d/b9371885084e4060ac157799e5c89b59 
> to temporary path on destination filesystem: 
> s3://bucket_name/data/default/z3_v2/c34a364639dab40857a8d18871a3f705/.tmp/3d69598daa9841f986da2341a5901444
> """
>
> I'm seeing the behavior using 's3://' or 's3n://'. Has anyone 
> experienced this or have advice?
>
> Thanks,
> Austin Heyne
>
> [1] 
> https://github.com/aheyne/geomesa/blob/b36507f4e999b295ebab8b1fb47a38f53e2a0e93/geomesa-hbase/geomesa-hbase-tools/src/main/scala/org/locationtech/geomesa/hbase/tools/ingest/HBaseBulkLoadCommand.scala#L47 
>

-- 
Austin L. Heyne