TEZ: Why there is no OOME in TEZ when intermediate files are kept in memory?

[Sachin Sabbarwal <sa...@gmail.com>]

Hi Guys
One quick question. I've read that when pig is running in TEZ mode,
intermediate files are not stored in HDFS, are kept in memory instead. So
that it's available to next task which is going to reuse our container. My
question is how does TEZ prevent a case of OOME, cause not writing to disk
and keeping everything in memory might cause OOME. Is there any threshold
after which files are written to HDFS? Am i missing something here?

Thanks
-- 
Sachin Sabbarwal
Linkedin: https://www.linkedin.com/profile?viewProfile=&key=95777265
Facebook: facebook.com/sachinsabbarwal
Quora: http://www.quora.com/Sachin-Sabbarwal
Blog: http://sachinsabbarwal.tumblr.com/

RE: Why there is no OOME in TEZ when intermediate files are kept in memory?

[Bikas Saha <bi...@hortonworks.com>]

As of today, the way Pig uses Tez, it specified disk as the destination of intermediate data. So its not kept in memory (though it may be in the OS buffer cache).
There are other artifacts (some of which are derived from intermediate inputs e.g. one side of a broadcast join) that are kept in memory in the container. These have a lifecycle associated with them which is set by Pig and are de-referenced (and GC’d) upon termination of that lifecycle.



From: Sachin Sabbarwal [mailto:sachinsabbarwal@gmail.com]
Sent: Tuesday, July 07, 2015 10:56 AM
To: user@tez.apache.org
Subject: TEZ: Why there is no OOME in TEZ when intermediate files are kept in memory?

Hi Guys
One quick question. I've read that when pig is running in TEZ mode, intermediate files are not stored in HDFS, are kept in memory instead. So that it's available to next task which is going to reuse our container. My question is how does TEZ prevent a case of OOME, cause not writing to disk and keeping everything in memory might cause OOME. Is there any threshold after which files are written to HDFS? Am i missing something here?

Thanks
--
Sachin Sabbarwal
Linkedin: https://www.linkedin.com/profile?viewProfile=&key=95777265
Facebook: facebook.com/sachinsabbarwal<http://facebook.com/sachinsabbarwal>
Quora: http://www.quora.com/Sachin-Sabbarwal
Blog: http://sachinsabbarwal.tumblr.com/