You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Beam JIRA Bot (Jira)" <ji...@apache.org> on 2020/09/27 17:08:02 UTC

[jira] [Assigned] (BEAM-10787) External sorter 2048MB memory limit should be Hadoop only

     [ https://issues.apache.org/jira/browse/BEAM-10787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Beam JIRA Bot reassigned BEAM-10787:
------------------------------------

    Assignee:     (was: Neville Li)

> External sorter 2048MB memory limit should be Hadoop only
> ---------------------------------------------------------
>
>                 Key: BEAM-10787
>                 URL: https://issues.apache.org/jira/browse/BEAM-10787
>             Project: Beam
>          Issue Type: Bug
>          Components: extensions-java-sorter
>    Affects Versions: 2.23.0
>            Reporter: Neville Li
>            Priority: P4
>              Labels: stale-assigned
>
> The 2048MB limit only applies when using Hadoop sequence file for external sort.
> It doesn't apply when sorting is entirely in memory since we're using a {{List<KV<byte[], byte[]>>}} to buffer data and {{long}} to keep track of total bytes. Same with native external sorter.
> https://github.com/apache/beam/pull/12659



--
This message was sent by Atlassian Jira
(v8.3.4#803005)