You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "CruiseCheung (Jira)" <ji...@apache.org> on 2021/03/30 03:47:00 UTC
[jira] [Created] (TEZ-4301) TezChlid OOM cause PipelineSorter may
create too many sort span
CruiseCheung created TEZ-4301:
---------------------------------
Summary: TezChlid OOM cause PipelineSorter may create too many sort span
Key: TEZ-4301
URL: https://issues.apache.org/jira/browse/TEZ-4301
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.9.2, 0.9.0
Environment: Previous sort span:
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=965738496, reserved.metasize=16777216
[TezChild] |impl.PipelinedSorter|: Reducer 24: Span0.length = 677, perItem = 1400122
[TezChild] |impl.PipelinedSorter|: Reducer 24: done sorting span=0, length=677, time=0
[TezChild] |impl.PipelinedSorter|: Reducer 24: Heap = SpanIterator<0:676> (span=Span[16777216,947882594]),
[TezChild] |impl.PipelinedSorter|: Reducer 24: Span131.length = 1, perItem = 442
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=276247288, reserved.metasize=16
[TezChild] |impl.PipelinedSorter|: Reducer 24: New Span132.length = 1, perItem = 442, counter:533341
[TezChild] |impl.PipelinedSorter|: Reducer 24: Span132.length = 1, perItem = 416
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=276246856, reserved.metasize=16
[TezChild] |impl.PipelinedSorter|: Reducer 24: New Span133.length = 1, perItem = 416, counter:533342
Reporter: CruiseCheung
When previous span perItem is much large than others, reserved.remaining() is large。This may create too many min sort span，cause now we use perItem length to deduce next sort span's metaSize.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)