You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Mehakmeet Singh (Jira)" <ji...@apache.org> on 2020/08/10 11:18:00 UTC
[jira] [Created] (HADOOP-17195) Intermittent OutOfMemory error
while performing hdfs CopyFromLocal to abfs
Mehakmeet Singh created HADOOP-17195:
----------------------------------------
Summary: Intermittent OutOfMemory error while performing hdfs CopyFromLocal to abfs
Key: HADOOP-17195
URL: https://issues.apache.org/jira/browse/HADOOP-17195
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Affects Versions: 3.3.0
Reporter: Mehakmeet Singh
Assignee: Mehakmeet Singh
OutOfMemory error due to new ThreadPools being made each time AbfsOutputStream is created. Since threadPool aren't limited a lot of data is loaded in buffer and thus it causes OutOfMemory error.
Possible fixes:
- Limit the number of ThreadCounts while performing hdfs copyFromLocal (Using -t property).
- Reducing OUTPUT_BUFFER_SIZE significantly which would limit the amount of buffer to be loaded in threads.
- Don't create new ThreadPools each time AbfsOutputStream is created and limit the number of ThreadPools each AbfsOutputStream could create.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org