You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Jonathan Eagles (JIRA)" <ji...@apache.org> on 2017/05/17 19:48:04 UTC
[jira] [Comment Edited] (TEZ-3732) Reduce Object size of
InputAttemptIdentifier and MapOutput for large jobs
[ https://issues.apache.org/jira/browse/TEZ-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014274#comment-16014274 ]
Jonathan Eagles edited comment on TEZ-3732 at 5/17/17 7:48 PM:
---------------------------------------------------------------
After:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 0x0000000800000000 base address and 0-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
{code}
{code}
Instantiated the sample instance via public org.apache.tez.runtime.library.common.InputAttemptIdentifier(int,int)
org.apache.tez.runtime.library.common.InputAttemptIdentifier object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 4 (object header) 01 00 00 00 (00000001 00000000 00000000 00000000) (1)
4 4 (object header) 00 00 00 00 (00000000 00000000 00000000 00000000) (0)
8 4 (object header) 28 c0 0c 00 (00101000 11000000 00001100 00000000) (835624)
12 4 int InputAttemptIdentifier.inputIdentifier 0
16 4 int InputAttemptIdentifier.attemptNumber 0
20 4 int InputAttemptIdentifier.spillEventId -1
24 1 boolean InputAttemptIdentifier.shared false
25 1 byte InputAttemptIdentifier.fetchTypeInfo 0
26 2 (alignment/padding gap)
28 4 java.lang.String InputAttemptIdentifier.pathComponent null
Instance size: 32 bytes
Space losses: 2 bytes internal + 0 bytes external = 2 bytes total
{code}
{code}
Failed to find matching constructor, falling back to class-only introspection.
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 12 (object header) N/A
12 4 int MapOutput.id N/A
16 1 boolean MapOutput.primaryMapOutput N/A
17 3 (alignment/padding gap)
20 4 org.apache.tez.runtime.library.common.InputAttemptIdentifier MapOutput.attemptIdentifier N/A
24 4 org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped MapOutput.callback N/A
28 4 (loss due to the next object alignment)
Instance size: 32 bytes
Space losses: 3 bytes internal + 4 bytes external = 7 bytes total
{code}
{code}
Instantiated the sample instance via org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$WaitMapOutput(org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$1)
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$WaitMapOutput object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 4 (object header) 01 00 00 00 (00000001 00000000 00000000 00000000) (1)
4 4 (object header) 00 00 00 00 (00000000 00000000 00000000 00000000) (0)
8 4 (object header) b8 dc 0c 00 (10111000 11011100 00001100 00000000) (842936)
12 4 int MapOutput.id 1
16 1 boolean MapOutput.primaryMapOutput false
17 3 (alignment/padding gap)
20 4 org.apache.tez.runtime.library.common.InputAttemptIdentifier MapOutput.attemptIdentifier null
24 4 org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped MapOutput.callback null
28 4 (loss due to the next object alignment)
Instance size: 32 bytes
Space losses: 3 bytes internal + 4 bytes external = 7 bytes total
{code}
{code}
Instantiated the sample instance via org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput(org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped,long,boolean,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$1)
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 4 (object header) 01 00 00 00 (00000001 00000000 00000000 00000000) (1)
4 4 (object header) 00 00 00 00 (00000000 00000000 00000000 00000000) (0)
8 4 (object header) 70 da 0c 00 (01110000 11011010 00001100 00000000) (842352)
12 4 int MapOutput.id 2
16 1 boolean MapOutput.primaryMapOutput false
17 3 (alignment/padding gap)
20 4 org.apache.tez.runtime.library.common.InputAttemptIdentifier MapOutput.attemptIdentifier null
24 4 org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped MapOutput.callback null
28 4 org.apache.hadoop.io.BoundedByteArrayOutputStream InMemoryMapOutput.byteStream (object)
Instance size: 32 bytes
Space losses: 3 bytes internal + 0 bytes external = 3 bytes total
{code}
{code}
Failed to find matching constructor, falling back to class-only introspection.
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$DiskDirectMapOutput object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 12 (object header) N/A
12 4 int MapOutput.id N/A
16 1 boolean MapOutput.primaryMapOutput N/A
17 3 (alignment/padding gap)
20 4 org.apache.tez.runtime.library.common.InputAttemptIdentifier MapOutput.attemptIdentifier N/A
24 4 org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped MapOutput.callback N/A
28 4 org.apache.hadoop.io.FileChunk DiskDirectMapOutput.outputPath N/A
Instance size: 32 bytes
Space losses: 3 bytes internal + 0 bytes external = 3 bytes total
{code}
{code}
Instantiated the sample instance via private org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$DiskMapOutput(org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped,long,org.apache.hadoop.fs.Path,long,boolean,org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.Path)
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$DiskMapOutput object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 4 (object header) 01 00 00 00 (00000001 00000000 00000000 00000000) (1)
4 4 (object header) 00 00 00 00 (00000000 00000000 00000000 00000000) (0)
8 4 (object header) 78 d5 0c 00 (01111000 11010101 00001100 00000000) (841080)
12 4 int MapOutput.id 5
16 1 boolean MapOutput.primaryMapOutput false
17 3 (alignment/padding gap)
20 4 org.apache.tez.runtime.library.common.InputAttemptIdentifier MapOutput.attemptIdentifier null
24 4 org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped MapOutput.callback null
28 4 org.apache.hadoop.fs.Path DiskMapOutput.tmpOutputPath null
32 4 org.apache.hadoop.io.FileChunk DiskMapOutput.outputPath (object)
36 4 java.io.OutputStream DiskMapOutput.disk null
Instance size: 40 bytes
Space losses: 3 bytes internal + 0 bytes external = 3 bytes total
{code}
was (Author: jeagles):
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 0x0000000800000000 base address and 0-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
{code}
{code}
Instantiated the sample instance via public org.apache.tez.runtime.library.common.InputAttemptIdentifier(int,int)
org.apache.tez.runtime.library.common.InputAttemptIdentifier object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 4 (object header) 01 00 00 00 (00000001 00000000 00000000 00000000) (1)
4 4 (object header) 00 00 00 00 (00000000 00000000 00000000 00000000) (0)
8 4 (object header) 28 c0 0c 00 (00101000 11000000 00001100 00000000) (835624)
12 4 int InputAttemptIdentifier.inputIdentifier 0
16 4 int InputAttemptIdentifier.attemptNumber 0
20 4 int InputAttemptIdentifier.spillEventId -1
24 1 boolean InputAttemptIdentifier.shared false
25 1 byte InputAttemptIdentifier.fetchTypeInfo 0
26 2 (alignment/padding gap)
28 4 java.lang.String InputAttemptIdentifier.pathComponent null
Instance size: 32 bytes
Space losses: 2 bytes internal + 0 bytes external = 2 bytes total
{code}
{code}
Failed to find matching constructor, falling back to class-only introspection.
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 12 (object header) N/A
12 4 int MapOutput.id N/A
16 1 boolean MapOutput.primaryMapOutput N/A
17 3 (alignment/padding gap)
20 4 org.apache.tez.runtime.library.common.InputAttemptIdentifier MapOutput.attemptIdentifier N/A
24 4 org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped MapOutput.callback N/A
28 4 (loss due to the next object alignment)
Instance size: 32 bytes
Space losses: 3 bytes internal + 4 bytes external = 7 bytes total
{code}
{code}
Instantiated the sample instance via org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$WaitMapOutput(org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$1)
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$WaitMapOutput object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 4 (object header) 01 00 00 00 (00000001 00000000 00000000 00000000) (1)
4 4 (object header) 00 00 00 00 (00000000 00000000 00000000 00000000) (0)
8 4 (object header) b8 dc 0c 00 (10111000 11011100 00001100 00000000) (842936)
12 4 int MapOutput.id 1
16 1 boolean MapOutput.primaryMapOutput false
17 3 (alignment/padding gap)
20 4 org.apache.tez.runtime.library.common.InputAttemptIdentifier MapOutput.attemptIdentifier null
24 4 org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped MapOutput.callback null
28 4 (loss due to the next object alignment)
Instance size: 32 bytes
Space losses: 3 bytes internal + 4 bytes external = 7 bytes total
{code}
{code}
Instantiated the sample instance via org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput(org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped,long,boolean,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$1)
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 4 (object header) 01 00 00 00 (00000001 00000000 00000000 00000000) (1)
4 4 (object header) 00 00 00 00 (00000000 00000000 00000000 00000000) (0)
8 4 (object header) 70 da 0c 00 (01110000 11011010 00001100 00000000) (842352)
12 4 int MapOutput.id 2
16 1 boolean MapOutput.primaryMapOutput false
17 3 (alignment/padding gap)
20 4 org.apache.tez.runtime.library.common.InputAttemptIdentifier MapOutput.attemptIdentifier null
24 4 org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped MapOutput.callback null
28 4 org.apache.hadoop.io.BoundedByteArrayOutputStream InMemoryMapOutput.byteStream (object)
Instance size: 32 bytes
Space losses: 3 bytes internal + 0 bytes external = 3 bytes total
{code}
{code}
Failed to find matching constructor, falling back to class-only introspection.
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$DiskDirectMapOutput object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 12 (object header) N/A
12 4 int MapOutput.id N/A
16 1 boolean MapOutput.primaryMapOutput N/A
17 3 (alignment/padding gap)
20 4 org.apache.tez.runtime.library.common.InputAttemptIdentifier MapOutput.attemptIdentifier N/A
24 4 org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped MapOutput.callback N/A
28 4 org.apache.hadoop.io.FileChunk DiskDirectMapOutput.outputPath N/A
Instance size: 32 bytes
Space losses: 3 bytes internal + 0 bytes external = 3 bytes total
{code}
{code}
Instantiated the sample instance via private org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$DiskMapOutput(org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped,long,org.apache.hadoop.fs.Path,long,boolean,org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.Path)
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$DiskMapOutput object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 4 (object header) 01 00 00 00 (00000001 00000000 00000000 00000000) (1)
4 4 (object header) 00 00 00 00 (00000000 00000000 00000000 00000000) (0)
8 4 (object header) 78 d5 0c 00 (01111000 11010101 00001100 00000000) (841080)
12 4 int MapOutput.id 5
16 1 boolean MapOutput.primaryMapOutput false
17 3 (alignment/padding gap)
20 4 org.apache.tez.runtime.library.common.InputAttemptIdentifier MapOutput.attemptIdentifier null
24 4 org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped MapOutput.callback null
28 4 org.apache.hadoop.fs.Path DiskMapOutput.tmpOutputPath null
32 4 org.apache.hadoop.io.FileChunk DiskMapOutput.outputPath (object)
36 4 java.io.OutputStream DiskMapOutput.disk null
Instance size: 40 bytes
Space losses: 3 bytes internal + 0 bytes external = 3 bytes total
{code}
> Reduce Object size of InputAttemptIdentifier and MapOutput for large jobs
> -------------------------------------------------------------------------
>
> Key: TEZ-3732
> URL: https://issues.apache.org/jira/browse/TEZ-3732
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Attachments: TEZ-3732.1.patch, TEZ-3732.2.patch
>
>
> Objects in 64bit java are 12bytes + member size aligned to 8 bytes
> InputAttemptIdentifier -> 33Bytes gets aligned up to 40 bytes
> This class is just one byte over the 32 byte alignment. Reducing object size by one byte can save 8 bytes per object.
> This is ~8MB savings for 1,000,000 inputs and ~80 MB savings for tasks with 10,000,000 inputs to fetch (Yes this is a real job)
> MapOutput -> 45 bytes gets aligned to 48 bytes
> This class can be sub-classed to avoid all sub-classes paying the object size cost for the other sub-classes
> Wait InMemory and DiskDirect -> 32 bytes
> Disk -> 40 bytes
> Total savings is harder to account for but more than the above case.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)