You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Malte Schwarzer <ms...@mieo.de> on 2014/11/09 22:54:18 UTC
How to make Flink to write less temporary files?
Hi,
I¹m having the problem that my tmp dir is running out of hard drive space
when running a map reduce job on a 1TB file. The job fails with "no space
left on device³ exception.
Probably the intermediate result set is getting too big. Is there a way to
avoid this? Or make flink to write less temporary files?
Thanks,
Malte
Re: How to make Flink to write less temporary files?
Posted by Stephan Ewen <se...@apache.org>.
It is hard to calculate, it very much depends on the job:
- Is it running filters that reduce the data volume early?
- It it possibly running operations that blow up the size of an
intermediate result?
I would in general assume to use as much temp space than the input data
size, unless you have a lot of RAM such that the system can process the job
completely in memory.
Unless you immediately filter the input data aggressively (you have a
highly selective filter function after the readTextFile(...) or so).
Stephan
On Mon, Nov 10, 2014 at 5:59 PM, Malte Schwarzer <ms...@mieo.de> wrote:
> What's the estimated amount of disk space for such a job? Or how can I
> calculate it?
>
> Malte
>
> Von: Stephan Ewen <se...@apache.org>
> Antworten an: <us...@flink.incubator.apache.org>
> Datum: Montag, 10. November 2014 11:22
> An: <us...@flink.incubator.apache.org>
> Betreff: Re: How to make Flink to write less temporary files?
>
> Hi!
>
> With 10 nodes and 25 GB on each node, you have 250 GB space to spill
> temporary files. You also seem to have roughly the same size in JVM Heap,
> out of which Flink can use roughly 2/3.
>
> When you process 1 TB, 250 GB JVM heap and 250 GB temp file space may not
> be enough, it is less than the initial data size.
>
> I think you need simply need more disk space for a job like that...
>
> Stephan
>
>
>
>
> On Mon, Nov 10, 2014 at 10:54 AM, Malte Schwarzer <ms...@mieo.de> wrote:
>
>> My blobStore fileds are small, but each *.channel file is around 170MB.
>> Before I start by Flink job I’ve 25GB free space available in my tmp-dir
>> and my taskmanager heap size is currently at 24GB. I’m using a cluster with
>> 10 nodes.
>>
>> Is this enough space to process a 1TB file?
>>
>> Von: Stephan Ewen <se...@apache.org>
>> Antworten an: <us...@flink.incubator.apache.org>
>> Datum: Montag, 10. November 2014 10:35
>> An: <us...@flink.incubator.apache.org>
>> Betreff: Re: How to make Flink to write less temporary files?
>>
>> I would assume that the blobStore fields are rather small (they are only
>> jar files so far).
>>
>> I would look for *.channel files, which are spilled intermediate results.
>> They can get pretty large for large jobs.
>>
>
>
Re: How to make Flink to write less temporary files?
Posted by Robert Metzger <rm...@apache.org>.
Hi,
maybe the tmp-dir's are mounted as ramdisks? Some Linux distributions have
this as a default configuration.
On Mon, Nov 10, 2014 at 2:22 AM, Stephan Ewen <se...@apache.org> wrote:
> Hi!
>
> With 10 nodes and 25 GB on each node, you have 250 GB space to spill
> temporary files. You also seem to have roughly the same size in JVM Heap,
> out of which Flink can use roughly 2/3.
>
> When you process 1 TB, 250 GB JVM heap and 250 GB temp file space may not
> be enough, it is less than the initial data size.
>
> I think you need simply need more disk space for a job like that...
>
> Stephan
>
>
>
>
> On Mon, Nov 10, 2014 at 10:54 AM, Malte Schwarzer <ms...@mieo.de> wrote:
>
>> My blobStore fileds are small, but each *.channel file is around 170MB.
>> Before I start by Flink job I’ve 25GB free space available in my tmp-dir
>> and my taskmanager heap size is currently at 24GB. I’m using a cluster with
>> 10 nodes.
>>
>> Is this enough space to process a 1TB file?
>>
>> Von: Stephan Ewen <se...@apache.org>
>> Antworten an: <us...@flink.incubator.apache.org>
>> Datum: Montag, 10. November 2014 10:35
>> An: <us...@flink.incubator.apache.org>
>> Betreff: Re: How to make Flink to write less temporary files?
>>
>> I would assume that the blobStore fields are rather small (they are only
>> jar files so far).
>>
>> I would look for *.channel files, which are spilled intermediate results.
>> They can get pretty large for large jobs.
>>
>
>
Re: How to make Flink to write less temporary files?
Posted by Malte Schwarzer <ms...@mieo.de>.
What's the estimated amount of disk space for such a job? Or how can I
calculate it?
Malte
Von: Stephan Ewen <se...@apache.org>
Antworten an: <us...@flink.incubator.apache.org>
Datum: Montag, 10. November 2014 11:22
An: <us...@flink.incubator.apache.org>
Betreff: Re: How to make Flink to write less temporary files?
Hi!
With 10 nodes and 25 GB on each node, you have 250 GB space to spill
temporary files. You also seem to have roughly the same size in JVM Heap,
out of which Flink can use roughly 2/3.
When you process 1 TB, 250 GB JVM heap and 250 GB temp file space may not be
enough, it is less than the initial data size.
I think you need simply need more disk space for a job like that...
Stephan
On Mon, Nov 10, 2014 at 10:54 AM, Malte Schwarzer <ms...@mieo.de> wrote:
> My blobStore fileds are small, but each *.channel file is around 170MB. Before
> I start by Flink job I¹ve 25GB free space available in my tmp-dir and my
> taskmanager heap size is currently at 24GB. I¹m using a cluster with 10 nodes.
>
> Is this enough space to process a 1TB file?
>
> Von: Stephan Ewen <se...@apache.org>
> Antworten an: <us...@flink.incubator.apache.org>
> Datum: Montag, 10. November 2014 10:35
> An: <us...@flink.incubator.apache.org>
> Betreff: Re: How to make Flink to write less temporary files?
>
>
> I would assume that the blobStore fields are rather small (they are only jar
> files so far).
>
> I would look for *.channel files, which are spilled intermediate results. They
> can get pretty large for large jobs.
Re: How to make Flink to write less temporary files?
Posted by Stephan Ewen <se...@apache.org>.
Hi!
With 10 nodes and 25 GB on each node, you have 250 GB space to spill
temporary files. You also seem to have roughly the same size in JVM Heap,
out of which Flink can use roughly 2/3.
When you process 1 TB, 250 GB JVM heap and 250 GB temp file space may not
be enough, it is less than the initial data size.
I think you need simply need more disk space for a job like that...
Stephan
On Mon, Nov 10, 2014 at 10:54 AM, Malte Schwarzer <ms...@mieo.de> wrote:
> My blobStore fileds are small, but each *.channel file is around 170MB.
> Before I start by Flink job I’ve 25GB free space available in my tmp-dir
> and my taskmanager heap size is currently at 24GB. I’m using a cluster with
> 10 nodes.
>
> Is this enough space to process a 1TB file?
>
> Von: Stephan Ewen <se...@apache.org>
> Antworten an: <us...@flink.incubator.apache.org>
> Datum: Montag, 10. November 2014 10:35
> An: <us...@flink.incubator.apache.org>
> Betreff: Re: How to make Flink to write less temporary files?
>
> I would assume that the blobStore fields are rather small (they are only
> jar files so far).
>
> I would look for *.channel files, which are spilled intermediate results.
> They can get pretty large for large jobs.
>
Re: How to make Flink to write less temporary files?
Posted by Malte Schwarzer <ms...@mieo.de>.
My blobStore fileds are small, but each *.channel file is around 170MB.
Before I start by Flink job I¹ve 25GB free space available in my tmp-dir and
my taskmanager heap size is currently at 24GB. I¹m using a cluster with 10
nodes.
Is this enough space to process a 1TB file?
Von: Stephan Ewen <se...@apache.org>
Antworten an: <us...@flink.incubator.apache.org>
Datum: Montag, 10. November 2014 10:35
An: <us...@flink.incubator.apache.org>
Betreff: Re: How to make Flink to write less temporary files?
I would assume that the blobStore fields are rather small (they are only jar
files so far).
I would look for *.channel files, which are spilled intermediate results.
They can get pretty large for large jobs.
Re: How to make Flink to write less temporary files?
Posted by Stephan Ewen <se...@apache.org>.
I would assume that the blobStore fields are rather small (they are only
jar files so far).
I would look for *.channel files, which are spilled intermediate results.
They can get pretty large for large jobs.
Re: How to make Flink to write less temporary files?
Posted by Till Rohrmann <ti...@gmail.com>.
Hi Flavio,
the BlobManager is supposed to delete the blobStore directories either
when the job finishes or when the cluster is shutdown. However, if the
system crashes, then it won't clean up the directories. Has your
system crashed recently and thus has not been shut down properly?
On Mon, Nov 10, 2014 at 9:37 AM, Ufuk Celebi <uc...@apache.org> wrote:
> @Flavio: These are used by the blob manager to distribute the JAR files in
> the cluster. I thought they were deleted when the task managers shut down.
> Maybe Till or Daniel can provide some information on the life cycle of blob
> manager objects.
>
> On Mon, Nov 10, 2014 at 9:24 AM, Flavio Pompermaier <po...@okkam.it>
> wrote:
>>
>> I also see a lot of blobStore-XXX directories in /tmp
>>
>>
>> On Mon, Nov 10, 2014 at 8:48 AM, Ufuk Celebi <uc...@apache.org> wrote:
>>>
>>> Hey Malte,
>>>
>>> thanks for reporting the issue. Did you change the default configuration?
>>> I've just checked and the default config hard codes the heap size for the
>>> task managers to 512 MB. In that case some of the algorithms will start
>>> spilling to disk earlier than necessary (assuming that you have more main
>>> memory available). You can either remove the config key
>>> "taskmanager.heap.mb" and let the JVM set a default max heap size or set the
>>> config key to a value appropriate for your machines/setup. Could you try
>>> this out and report back?
>>>
>>> Regarding the temp directories: How much disk space is available before
>>> running your program? I remember a user reporting the same issue, because of
>>> some old files lingering around in the temp directories.
>>>
>>> – Ufuk
>>>
>>>
>>> On Sun, Nov 9, 2014 at 10:54 PM, Malte Schwarzer <ms...@mieo.de> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I’m having the problem that my tmp dir is running out of hard drive
>>>> space when running a map reduce job on a 1TB file. The job fails with "no
>>>> space left on device“ exception.
>>>>
>>>> Probably the intermediate result set is getting too big. Is there a way
>>>> to avoid this? Or make flink to write less temporary files?
>>>>
>>>> Thanks,
>>>> Malte
>>>>
>>>>
>>>
>>
>>
>
Re: How to make Flink to write less temporary files?
Posted by Ufuk Celebi <uc...@apache.org>.
@Flavio: These are used by the blob manager to distribute the JAR files in
the cluster. I thought they were deleted when the task managers shut down.
Maybe Till or Daniel can provide some information on the life cycle of blob
manager objects.
On Mon, Nov 10, 2014 at 9:24 AM, Flavio Pompermaier <po...@okkam.it>
wrote:
> I also see a lot of blobStore-XXX directories in /tmp
>
>
> On Mon, Nov 10, 2014 at 8:48 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
>> Hey Malte,
>>
>> thanks for reporting the issue. Did you change the default configuration?
>> I've just checked and the default config hard codes the heap size for the
>> task managers to 512 MB. In that case some of the algorithms will start
>> spilling to disk earlier than necessary (assuming that you have more main
>> memory available). You can either remove the config key "
>> taskmanager.heap.mb" and let the JVM set a default max heap size or set
>> the config key to a value appropriate for your machines/setup. Could you
>> try this out and report back?
>>
>> Regarding the temp directories: How much disk space is available before
>> running your program? I remember a user reporting the same issue, because
>> of some old files lingering around in the temp directories.
>>
>> – Ufuk
>>
>>
>> On Sun, Nov 9, 2014 at 10:54 PM, Malte Schwarzer <ms...@mieo.de> wrote:
>>
>>> Hi,
>>>
>>> I’m having the problem that my tmp dir is running out of hard drive
>>> space when running a map reduce job on a 1TB file. The job fails with "no
>>> space left on device“ exception.
>>>
>>> Probably the intermediate result set is getting too big. Is there a way
>>> to avoid this? Or make flink to write less temporary files?
>>>
>>> Thanks,
>>> Malte
>>>
>>>
>>>
>>
>
>
Re: How to make Flink to write less temporary files?
Posted by Flavio Pompermaier <po...@okkam.it>.
I also see a lot of blobStore-XXX directories in /tmp
On Mon, Nov 10, 2014 at 8:48 AM, Ufuk Celebi <uc...@apache.org> wrote:
> Hey Malte,
>
> thanks for reporting the issue. Did you change the default configuration?
> I've just checked and the default config hard codes the heap size for the
> task managers to 512 MB. In that case some of the algorithms will start
> spilling to disk earlier than necessary (assuming that you have more main
> memory available). You can either remove the config key "
> taskmanager.heap.mb" and let the JVM set a default max heap size or set
> the config key to a value appropriate for your machines/setup. Could you
> try this out and report back?
>
> Regarding the temp directories: How much disk space is available before
> running your program? I remember a user reporting the same issue, because
> of some old files lingering around in the temp directories.
>
> – Ufuk
>
>
> On Sun, Nov 9, 2014 at 10:54 PM, Malte Schwarzer <ms...@mieo.de> wrote:
>
>> Hi,
>>
>> I’m having the problem that my tmp dir is running out of hard drive
>> space when running a map reduce job on a 1TB file. The job fails with "no
>> space left on device“ exception.
>>
>> Probably the intermediate result set is getting too big. Is there a way
>> to avoid this? Or make flink to write less temporary files?
>>
>> Thanks,
>> Malte
>>
>>
>>
>
Re: How to make Flink to write less temporary files?
Posted by Ufuk Celebi <uc...@apache.org>.
Hey Malte,
thanks for reporting the issue. Did you change the default configuration?
I've just checked and the default config hard codes the heap size for the
task managers to 512 MB. In that case some of the algorithms will start
spilling to disk earlier than necessary (assuming that you have more main
memory available). You can either remove the config key "taskmanager.heap.mb
" and let the JVM set a default max heap size or set the config key to a
value appropriate for your machines/setup. Could you try this out and
report back?
Regarding the temp directories: How much disk space is available before
running your program? I remember a user reporting the same issue, because
of some old files lingering around in the temp directories.
– Ufuk
On Sun, Nov 9, 2014 at 10:54 PM, Malte Schwarzer <ms...@mieo.de> wrote:
> Hi,
>
> I’m having the problem that my tmp dir is running out of hard drive space
> when running a map reduce job on a 1TB file. The job fails with "no space
> left on device“ exception.
>
> Probably the intermediate result set is getting too big. Is there a way to
> avoid this? Or make flink to write less temporary files?
>
> Thanks,
> Malte
>
>
>