You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Amir Youssefi (JIRA)" <ji...@apache.org> on 2008/03/21 22:07:25 UTC
[jira] Created: (PIG-166) Disk Full
Disk Full
---------
Key: PIG-166
URL: https://issues.apache.org/jira/browse/PIG-166
Project: Pig
Issue Type: Bug
Reporter: Amir Youssefi
Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-166) Disk Full
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584207#action_12584207 ]
Olga Natkovich commented on PIG-166:
------------------------------------
Pi,
At this point I think we are bound by CPU rather than disk writes (especially as we do local writes) but over time that might change. Also, I think we could get fairly high compression rate and save significant amount of space. So I do agree that we need to look into this.
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-166) Disk Full
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12583719#action_12583719 ]
Alan Gates commented on PIG-166:
--------------------------------
Pi,
This is in response to your comments about garbage collected temp files. Currently, we mark temp files from bag spills as deleteOnExit. So unless the JVM crashes or is killed via kill -9 or similar mechanism, we do clean up our tmp files. The issue that Amir is addressing here is when running a single pig job fills up the disks.
AFAIK, if people's jobs generate so much data that we can't even contain it on the disk then our only hope is to find a way to better parallelize the problem. That will not always be possible. But as Amir points out, when we do fail we shouldn't take down the node with us. I think that's really the focus of this bug.
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-166) Disk Full
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588356#action_12588356 ]
Pi Song commented on PIG-166:
-----------------------------
A cluster can have a large number of machines and not all of them are likely to have the same amount of free disk space. I think a more practical solution would be to be able to set up the low water mark of available disk space where temp files are no longer allowed to be created.
A naive Implementation would be to keep checking the current available disk space over time but that will degrade the execution performance too much. A more clever way is to keep the initial available disk space and maintain that variable against space used by our temp files but this way is not practical because to make it work correctly we have to make sure there is only our process utilizing disk at any time.
Any thought? Idea?
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
> Attachments: PIG-166_v1.patch
>
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-166) Disk Full
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584687#action_12584687 ]
Alan Gates commented on PIG-166:
--------------------------------
Based on Pi's observation that deleteOnExit is too lazy and we delete the bag spill files much sooner (when the bag is finalized), I checked the code, and we are already doing that. The DataBag.finalize() method walks the list of spill files and deletes them.
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-166) Disk Full
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12583742#action_12583742 ]
Olga Natkovich commented on PIG-166:
------------------------------------
Couple of comments:
(1) We need to validate that we are spilling correctly, like not spilling the same data multiple times
(2) If this is legitimate data, we need to look at compressing it
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-166) Disk Full
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586075#action_12586075 ]
Pi Song commented on PIG-166:
-----------------------------
PS. This is controlling based on "current temp space". If you want to change to control based on "current disk space left", only a small change is needed to the patch.
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
> Attachments: PIG-166_v1.patch
>
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-166) Disk Full
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581926#action_12581926 ]
Pi Song commented on PIG-166:
-----------------------------
Also relating to bag spilling problem that we usually have a lot of small bags (too many small files=slow on most filesystems), I'm thinking about having the file manager managing packing of small files into big files for better performance.
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-166) Disk Full
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pi Song updated PIG-166:
------------------------
Patch Info: [Patch Available]
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
> Attachments: PIG-166_v1.patch
>
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-166) Disk Full
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585080#action_12585080 ]
Pi Song commented on PIG-166:
-----------------------------
Can we do something like this?
{code}
public void cleanUpTempFiles() {
// do clean up temp files here
alreadyClean = true ;
}
protected void finalize() {
if (!alreadyClean) {
cleanUpTempFiles() ;
}
}
{code}
My observation is for finalize() you have to wait for GC to do it. If you know you don't need it, why don't just clean it right away (optional)
1.GC is non-determistic
2. Your bag might live in older generation therefore unlikely to get clean-up
3. If the clean-up in finalize() takes too much time, GC will have to wait for too long therefore under memory pressure it may not efficiently free more memory.
PS. The main issue is to prevent disk full. I think this is still a part of the solution.
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-166) Disk Full
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12583960#action_12583960 ]
Pi Song commented on PIG-166:
-----------------------------
Olga,
2) Compressing data before writing to disk sometimes can result in a significant performance gain (As discussed in some RDBMS papers) given a write-once scenario and your task is not yet fully utilizing CPU. I think this area is worth exploring.
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-166) Disk Full
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585164#action_12585164 ]
Olga Natkovich commented on PIG-166:
------------------------------------
I think it makes sense. The issue is do we currently now when a bug is going out of scope?
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-166) Disk Full
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586073#action_12586073 ]
Pi Song commented on PIG-166:
-----------------------------
I have implemented a minimalist temp file manager (probably not minimalist but not as much as I expected) just to get out of the current problem quickly
- "temp.limit" can be set via system variable or .pigrc. By default, it is set to LONG.MAX_VALUE.
- We keep track of all temp space usage by hooking up when temp streams are closed (That's where the actual file size is).
- If the execution engine is exceeding disk space limit and a temp file is to be created, the temp file manager will try to free up more space first. If still there is no space, a RuntimeException will be thrown, causing the execution to stop.
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-166) Disk Full
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pi Song updated PIG-166:
------------------------
Attachment: PIG-166_v1.patch
The patch. All unit test passed.
Feel free to take my patch and modify if you have a good technical reason to do so.
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
> Attachments: PIG-166_v1.patch
>
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-166) Disk Full
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12583958#action_12583958 ]
Pi Song commented on PIG-166:
-----------------------------
Alan,
You're right that temp files are already taken care of but deleteOnExit() only does clean up temp files before JVM terminates (if no crash). If the life cycle of those files are known, you can delete them in the middle of processing to reclaim more space and the execution therefore can go more off-limit.
What I implied here before is also that CreateTempFile() is not good because you cannot control the limit of space you use so it's good to have a temp file manager that manages space usage .
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-166) Disk Full
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581919#action_12581919 ]
Pi Song commented on PIG-166:
-----------------------------
Amir,
So the problem is mainly due to pig hogging disk space right?
As I said before, I think we should have some kind of TempFileManager. A simple one shouldn't take too much time to implement.
(Copied from what I've already posted before)
- Rely on a temp folder and temp files on top of file systems. (Or should we rely and wait for Hadoop to have?)
- User can customize location and size limit.
- The folder is clean-up every time the system restarts (comparable to when pig restarts, not hadoop restarts so this can be more frequent)
- The old temp files are removed when the system running low on temp space
- Temp files where lifecycle is explicitly known may be marked for collection at appropriate time (like the concept of GC)
I can start working on this if people agree.
> Disk Full
> ---------
>
> Key: PIG-166
> URL: https://issues.apache.org/jira/browse/PIG-166
> Project: Pig
> Issue Type: Bug
> Reporter: Amir Youssefi
>
> Occasionally spilling fills up (all) hard drive(s) on a Data Node and crashes Task Tracker (and other processes) on that node. We need to have a safety net and fail the task before crashing happens (and more).
> In Pig + Hadoop setting, Task Trackers get Black Listed. And Pig console gets stock at a percentage without returning nodes to cluster. I talked to Hadoop team to explore Max Percentage idea. Nodes running into this problem get into permanent problems and manual cleaning by administrator is necessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.