You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Muhammad Samir Khan (JIRA)" <ji...@apache.org> on 2017/08/03 20:44:00 UTC
[jira] [Commented] (TEZ-3813) Reduce Object size of
MemoryFetchedInput for large jobs
[ https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113451#comment-16113451 ]
Muhammad Samir Khan commented on TEZ-3813:
------------------------------------------
*JOL Dump:*
+Before:+
Internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 4 (object header) 01 00 00 00 (00000001 00000000 00000000 00000000) (1)
4 4 (object header) 00 00 00 00 (00000000 00000000 00000000 00000000) (0)
8 4 (object header) 7a 12 01 f8 (01111010 00010010 00000001 11111000) (-134147462)
12 4 int FetchedInput.id 0
16 8 long FetchedInput.actualSize 0
24 8 long FetchedInput.compressedSize 0
32 4 org.apache.tez.runtime.library.common.InputAttemptIdentifier FetchedInput.inputAttemptIdentifier null
36 4 org.apache.tez.runtime.library.common.shuffle.FetchedInput.Type FetchedInput.type (object)
40 4 org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback FetchedInput.callback null
44 4 org.apache.tez.runtime.library.common.shuffle.FetchedInput.State FetchedInput.state (object)
48 4 org.apache.hadoop.io.BoundedByteArrayOutputStream MemoryFetchedInput.byteStream (object)
52 4 (loss due to the next object alignment)
Instance size: 56 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total
{code}
Footprint:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput@215be6bbd footprint:
COUNT AVG SUM DESCRIPTION
1 16 16 [B
2 32 64 [C
2 24 48 java.lang.String
1 32 32 org.apache.hadoop.io.BoundedByteArrayOutputStream
1 24 24 org.apache.tez.runtime.library.common.shuffle.FetchedInput$State
1 24 24 org.apache.tez.runtime.library.common.shuffle.FetchedInput$Type
1 56 56 org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput
9 264 (total)
{code}
+After:+
Internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 4 (object header) 01 00 00 00 (00000001 00000000 00000000 00000000) (1)
4 4 (object header) 00 00 00 00 (00000000 00000000 00000000 00000000) (0)
8 4 (object header) 7a 12 01 f8 (01111010 00010010 00000001 11111000) (-134147462)
12 4 int FetchedInput.id 0
16 8 long FetchedInput.actualSize 0
24 8 long FetchedInput.compressedSize 0
32 4 org.apache.tez.runtime.library.common.InputAttemptIdentifier FetchedInput.inputAttemptIdentifier null
36 4 org.apache.tez.runtime.library.common.shuffle.FetchedInput.Type FetchedInput.type (object)
40 4 org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback FetchedInput.callback null
44 4 org.apache.tez.runtime.library.common.shuffle.FetchedInput.State FetchedInput.state (object)
48 4 byte[] MemoryFetchedInput.byteArray []
52 4 (loss due to the next object alignment)
Instance size: 56 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total
{code}
Footprint:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput@5dfcfeced footprint:
COUNT AVG SUM DESCRIPTION
1 16 16 [B
2 32 64 [C
2 24 48 java.lang.String
1 24 24 org.apache.tez.runtime.library.common.shuffle.FetchedInput$State
1 24 24 org.apache.tez.runtime.library.common.shuffle.FetchedInput$Type
1 56 56 org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput
8 232 (total)
{code}
> Reduce Object size of MemoryFetchedInput for large jobs
> -------------------------------------------------------
>
> Key: TEZ-3813
> URL: https://issues.apache.org/jira/browse/TEZ-3813
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Muhammad Samir Khan
> Assignee: Muhammad Samir Khan
> Attachments: TEZ-3813.001.patch
>
>
> Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a BoundedByteArrayOutputStream that is not used (only the underlying byte[] is used).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)