[jira] [Commented] (SPARK-1154) Spark fills up disk with app-* folders

["Mingyu Kim (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999562#comment-13999562 ] 

Mingyu Kim commented on SPARK-1154:
-----------------------------------

I looked at the commit, and it seems like it wipes out app-* based on the last modification time of the directory itself. Because the modification time of a directory only changes when a child is added or removed, this may wipe out the app-* directory of a running Spark application if it has been running for more than TTL (unless new files/jars are added to the app directory once every while). I believe it should check the latest modification time of all the descendents of the app-* directory to decide whether to delete it or not. Am I mistaken?

> Spark fills up disk with app-* folders
> --------------------------------------
>
>                 Key: SPARK-1154
>                 URL: https://issues.apache.org/jira/browse/SPARK-1154
>             Project: Spark
>          Issue Type: Improvement
>          Components: Deploy
>            Reporter: Evan Chan
>            Assignee: Evan Chan
>            Priority: Critical
>              Labels: starter
>             Fix For: 1.0.0
>
>
> Current version of Spark fills up the disk with many app-* folders:
> $ ls /var/lib/spark
> app-20140210022347-0597  app-20140212173327-0627  app-20140218154110-0657  app-20140225232537-0017  app-20140225233548-0047
> app-20140210022407-0598  app-20140212173347-0628  app-20140218154130-0658  app-20140225232551-0018  app-20140225233556-0048
> app-20140210022427-0599  app-20140212173754-0629  app-20140218164232-0659  app-20140225232611-0019  app-20140225233603-0049
> app-20140210022447-0600  app-20140212182235-0630  app-20140218165133-0660  app-20140225232802-0020  app-20140225233610-0050
> app-20140210022508-0601  app-20140212182256-0631  app-20140218165148-0661  app-20140225232822-0021  app-20140225233617-0051
> app-20140210022528-0602  app-20140213000014-0632  app-20140218165225-0662  app-20140225232940-0022  app-20140225233624-0052
> app-20140211024356-0603  app-20140213002026-0633  app-20140218165249-0663  app-20140225233002-0023  app-20140225233631-0053
> app-20140211024417-0604  app-20140213154948-0634  app-20140218172030-0664  app-20140225233056-0024  app-20140225233725-0054
> app-20140211024437-0605  app-20140213171810-0635  app-20140218193853-0665  app-20140225233108-0025  app-20140225233731-0055
> app-20140211024457-0606  app-20140213193637-0636  app-20140218194442-0666  app-20140225233124-0026  app-20140225233733-0056
> app-20140211024517-0607  app-20140214011513-0637  app-20140218194746-0667  app-20140225233133-0027  app-20140225233734-0057
> app-20140211024538-0608  app-20140214012151-0638  app-20140218194822-0668  app-20140225233147-0028  app-20140225233749-0058
> app-20140211193443-0609  app-20140214013134-0639  app-20140218212317-0669  app-20140225233208-0029  app-20140225233759-0059
> app-20140211195210-0610  app-20140214013332-0640  app-20140225180142-0000  app-20140225233215-0030  app-20140225233809-0060
> app-20140211213935-0611  app-20140214013642-0641  app-20140225180411-0001  app-20140225233224-0031  app-20140225233828-0061
> app-20140211214227-0612  app-20140214014246-0642  app-20140225180431-0002  app-20140225233232-0032  app-20140225234719-0062
> app-20140211215317-0613  app-20140214014607-0643  app-20140225180452-0003  app-20140225233239-0033  app-20140226032845-0063
> app-20140211224601-0614  app-20140214184943-0644  app-20140225180512-0004  app-20140225233320-0034  app-20140226033004-0064
> app-20140212022206-0615  app-20140214185118-0645  app-20140225180533-0005  app-20140225233328-0035  app-20140226033119-0065
> app-20140212022226-0616  app-20140214185851-0646  app-20140225180553-0006  app-20140225233354-0036  app-20140226033334-0066
> app-20140212022246-0617  app-20140214222856-0647  app-20140225181115-0007  app-20140225233402-0037  app-20140226033354-0067
> app-20140212043704-0618  app-20140214231312-0648  app-20140225181244-0008  app-20140225233409-0038  app-20140226033538-0068
> app-20140212043724-0619  app-20140214231434-0649  app-20140225182051-0009  app-20140225233416-0039  app-20140226033826-0069
> app-20140212043745-0620  app-20140214231542-0650  app-20140225183009-0010  app-20140225233426-0040  app-20140226034002-0070
> app-20140212044016-0621  app-20140214231616-0651  app-20140225184133-0011  app-20140225233432-0041  app-20140226034053-0071
> app-20140212044203-0622  app-20140214233016-0652  app-20140225184318-0012  app-20140225233439-0042  app-20140226034234-0072
> app-20140212044224-0623  app-20140214233037-0653  app-20140225184709-0013  app-20140225233447-0043  app-20140226034426-0073
> app-20140212045034-0624  app-20140218153242-0654  app-20140225184844-0014  app-20140225233526-0044  app-20140226034447-0074
> app-20140212045119-0625  app-20140218153341-0655  app-20140225190051-0015  app-20140225233534-0045
> app-20140212173310-0626  app-20140218153442-0656  app-20140225232516-0016  app-20140225233540-0046
> This problem is particularly bad if you have a whole bunch of fast jobs.   
> Also what makes the problem worse is that any jars for jobs is downloaded into the app-* folder, so that fills up the disk particularly fast.
> I would like to propose two things:
> 1) Spark should have a cleanup thread (or actor) which periodically removes old app-* folders;   This should not be the responsibility of people deploying Spark.
> 2) The download of jars should not go to each app-* folder.  This wastes a huge amount of space because most jobs use the same jars.  Maybe I can open a separate ticket for this.



--
This message was sent by Atlassian JIRA
(v6.2#6252)