You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Udit Mehrotra (Jira)" <ji...@apache.org> on 2021/01/15 00:25:00 UTC
[jira] [Updated] (HUDI-1529) Spark-SQL drvier runs out of memory
when metadata table is enabled
[ https://issues.apache.org/jira/browse/HUDI-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Udit Mehrotra updated HUDI-1529:
--------------------------------
Description:
When testing a large dataset around 1.2TB data and around 20k files, we notice an issue where the spark driver would always run out of memory, when running queries with use of metadata table *enabled*. The OOM would happen on any query, even if it was touching a single partition, and was happening in the *split generation* phase before any tasks would start executing.
Upon analyzing the heap dump, it was analyzed that input format was generating *millions of splits for every single file*. Upon further analysis of the code path, it was found that the root cause was because *metadata enabled* code was ignoring the *blockSize* when returning *FileStatus* objects and setting it to *0*. Spark by itself does not set any value for the property:
{code:java}
mapreduce.input.fileinputformat.split.minsize
{code}
As a result *minSize* ends up being 1, and with block size as 0 it cause input format to *generate splits of size 1 bytes***** because of the logic here:
[https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L417]
This ends up in exponential file split objects being creating, causing driver to run out of memory.
> Spark-SQL drvier runs out of memory when metadata table is enabled
> ------------------------------------------------------------------
>
> Key: HUDI-1529
> URL: https://issues.apache.org/jira/browse/HUDI-1529
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Performance, Spark Integration
> Reporter: Udit Mehrotra
> Assignee: Udit Mehrotra
> Priority: Major
>
> When testing a large dataset around 1.2TB data and around 20k files, we notice an issue where the spark driver would always run out of memory, when running queries with use of metadata table *enabled*. The OOM would happen on any query, even if it was touching a single partition, and was happening in the *split generation* phase before any tasks would start executing.
> Upon analyzing the heap dump, it was analyzed that input format was generating *millions of splits for every single file*. Upon further analysis of the code path, it was found that the root cause was because *metadata enabled* code was ignoring the *blockSize* when returning *FileStatus* objects and setting it to *0*. Spark by itself does not set any value for the property:
> {code:java}
> mapreduce.input.fileinputformat.split.minsize
> {code}
> As a result *minSize* ends up being 1, and with block size as 0 it cause input format to *generate splits of size 1 bytes***** because of the logic here:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L417]
> This ends up in exponential file split objects being creating, causing driver to run out of memory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)