You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2009/07/24 21:25:14 UTC
[jira] Created: (HIVE-690) ScriptOperator still times out
ScriptOperator still times out
------------------------------
Key: HIVE-690
URL: https://issues.apache.org/jira/browse/HIVE-690
Project: Hadoop Hive
Issue Type: Bug
Reporter: Zheng Shao
Assignee: Zheng Shao
Fix For: 0.4.0
The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
{code}
Integer exp_interval = null;
int exp_int;
exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
if (exp_interval != null)
exp_int = exp_interval.intValue() / 2;
else
exp_int = 300000;
rpTimer = new Timer(true);
rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-690) ScriptOperator still times out
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735633#action_12735633 ]
Namit Jain commented on HIVE-690:
---------------------------------
By default, the auto progress is false, which means that the timer would also be off.
It is a change from the existing behavior - is that acceptable ?
> ScriptOperator still times out
> ------------------------------
>
> Key: HIVE-690
> URL: https://issues.apache.org/jira/browse/HIVE-690
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Blocker
> Fix For: 0.4.0
>
> Attachments: HIVE-690.1.patch, HIVE-690.2.patch
>
>
> The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
> {code}
> Integer exp_interval = null;
> int exp_int;
> exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
> if (exp_interval != null)
> exp_int = exp_interval.intValue() / 2;
> else
> exp_int = 300000;
> rpTimer = new Timer(true);
> rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-690) ScriptOperator still times out
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735721#action_12735721 ]
Namit Jain commented on HIVE-690:
---------------------------------
+1
Running tests right now
> ScriptOperator still times out
> ------------------------------
>
> Key: HIVE-690
> URL: https://issues.apache.org/jira/browse/HIVE-690
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Blocker
> Fix For: 0.4.0
>
> Attachments: HIVE-690.1.patch, HIVE-690.2.patch
>
>
> The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
> {code}
> Integer exp_interval = null;
> int exp_int;
> exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
> if (exp_interval != null)
> exp_int = exp_interval.intValue() / 2;
> else
> exp_int = 300000;
> rpTimer = new Timer(true);
> rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-690) ScriptOperator still times out
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-690:
----------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Zheng
> ScriptOperator still times out
> ------------------------------
>
> Key: HIVE-690
> URL: https://issues.apache.org/jira/browse/HIVE-690
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Blocker
> Fix For: 0.4.0
>
> Attachments: HIVE-690.1.patch, HIVE-690.2.patch
>
>
> The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
> {code}
> Integer exp_interval = null;
> int exp_int;
> exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
> if (exp_interval != null)
> exp_int = exp_interval.intValue() / 2;
> else
> exp_int = 300000;
> rpTimer = new Timer(true);
> rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-690) ScriptOperator still times out
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735132#action_12735132 ]
Namit Jain commented on HIVE-690:
---------------------------------
This might explain some timeouts
> ScriptOperator still times out
> ------------------------------
>
> Key: HIVE-690
> URL: https://issues.apache.org/jira/browse/HIVE-690
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
>
> The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
> {code}
> Integer exp_interval = null;
> int exp_int;
> exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
> if (exp_interval != null)
> exp_int = exp_interval.intValue() / 2;
> else
> exp_int = 300000;
> rpTimer = new Timer(true);
> rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-690) ScriptOperator still times out
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-690:
----------------------------
Attachment: HIVE-690.1.patch
This patch fixes the problem.
It also changes the naming convention to the Java naming convention.
> ScriptOperator still times out
> ------------------------------
>
> Key: HIVE-690
> URL: https://issues.apache.org/jira/browse/HIVE-690
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-690.1.patch
>
>
> The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
> {code}
> Integer exp_interval = null;
> int exp_int;
> exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
> if (exp_interval != null)
> exp_int = exp_interval.intValue() / 2;
> else
> exp_int = 300000;
> rpTimer = new Timer(true);
> rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-690) ScriptOperator still times out
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735685#action_12735685 ]
Zheng Shao commented on HIVE-690:
---------------------------------
The old timer code was never useful because the timeout is the same as the expiry period, so the behavior didn't change actually.
I think this is what we want. If we set it to true by default, it is very easy for some programs to enter infinite loops and never finish.
We should enable it only with users' explicit command "set hive.script.auto.progress=true;".
> ScriptOperator still times out
> ------------------------------
>
> Key: HIVE-690
> URL: https://issues.apache.org/jira/browse/HIVE-690
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Blocker
> Fix For: 0.4.0
>
> Attachments: HIVE-690.1.patch, HIVE-690.2.patch
>
>
> The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
> {code}
> Integer exp_interval = null;
> int exp_int;
> exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
> if (exp_interval != null)
> exp_int = exp_interval.intValue() / 2;
> else
> exp_int = 300000;
> rpTimer = new Timer(true);
> rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-690) ScriptOperator still times out
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-690:
----------------------------
Attachment: HIVE-690.2.patch
This patch fixes the current timer approach (with a configurable option), and adds the capability to do "reporter.progress()" for each stderr line (but no more frequently than once per minute).
> ScriptOperator still times out
> ------------------------------
>
> Key: HIVE-690
> URL: https://issues.apache.org/jira/browse/HIVE-690
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Blocker
> Fix For: 0.4.0
>
> Attachments: HIVE-690.1.patch, HIVE-690.2.patch
>
>
> The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
> {code}
> Integer exp_interval = null;
> int exp_int;
> exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
> if (exp_interval != null)
> exp_int = exp_interval.intValue() / 2;
> else
> exp_int = 300000;
> rpTimer = new Timer(true);
> rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-690) ScriptOperator still times out
Posted by "Venky Iyer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735169#action_12735169 ]
Venky Iyer commented on HIVE-690:
---------------------------------
I believe that it is sufficient to track stderr output to see if the process is alive.
> ScriptOperator still times out
> ------------------------------
>
> Key: HIVE-690
> URL: https://issues.apache.org/jira/browse/HIVE-690
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-690.1.patch
>
>
> The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
> {code}
> Integer exp_interval = null;
> int exp_int;
> exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
> if (exp_interval != null)
> exp_int = exp_interval.intValue() / 2;
> else
> exp_int = 300000;
> rpTimer = new Timer(true);
> rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-690) ScriptOperator still times out
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735231#action_12735231 ]
Ashish Thusoo commented on HIVE-690:
------------------------------------
Instead of asking the users to have a timer in python to send information to stderr a better approach (and the one that we can strongly recommend) would be to check some stats that are maintained by the main loop and only report stderr when those stats have changed. With the current approach that is just not possible, so application errors can easily be amplified. That was the concern that Joy had raised with the previous fix (after it got checked in).
> ScriptOperator still times out
> ------------------------------
>
> Key: HIVE-690
> URL: https://issues.apache.org/jira/browse/HIVE-690
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-690.1.patch
>
>
> The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
> {code}
> Integer exp_interval = null;
> int exp_int;
> exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
> if (exp_interval != null)
> exp_int = exp_interval.intValue() / 2;
> else
> exp_int = 300000;
> rpTimer = new Timer(true);
> rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-690) ScriptOperator still times out
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-690:
----------------------------
Priority: Blocker (was: Major)
Escalating this issue to blocker since there is currently no way for a user script to run over 10 minutes without creating an output row.
> ScriptOperator still times out
> ------------------------------
>
> Key: HIVE-690
> URL: https://issues.apache.org/jira/browse/HIVE-690
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Blocker
> Fix For: 0.4.0
>
> Attachments: HIVE-690.1.patch
>
>
> The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
> {code}
> Integer exp_interval = null;
> int exp_int;
> exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
> if (exp_interval != null)
> exp_int = exp_interval.intValue() / 2;
> else
> exp_int = 300000;
> rpTimer = new Timer(true);
> rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-690) ScriptOperator still times out
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735229#action_12735229 ]
Zheng Shao commented on HIVE-690:
---------------------------------
The major goal of status reporting is probably more for the detecting dead process/dead node than for infinite loops.
Also, users will probably just output stderr messages using timers in their python code.
If that's the case, it does not prevent our users' program from infinite loop.
I think we should check this fix in first (after all it's so simple a fix). Thoughts?
> ScriptOperator still times out
> ------------------------------
>
> Key: HIVE-690
> URL: https://issues.apache.org/jira/browse/HIVE-690
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-690.1.patch
>
>
> The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
> {code}
> Integer exp_interval = null;
> int exp_int;
> exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
> if (exp_interval != null)
> exp_int = exp_interval.intValue() / 2;
> else
> exp_int = 300000;
> rpTimer = new Timer(true);
> rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-690) ScriptOperator still times out
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-690:
----------------------------
Status: Patch Available (was: Open)
> ScriptOperator still times out
> ------------------------------
>
> Key: HIVE-690
> URL: https://issues.apache.org/jira/browse/HIVE-690
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-690.1.patch
>
>
> The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
> {code}
> Integer exp_interval = null;
> int exp_int;
> exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
> if (exp_interval != null)
> exp_int = exp_interval.intValue() / 2;
> else
> exp_int = 300000;
> rpTimer = new Timer(true);
> rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-690) ScriptOperator still times out
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735152#action_12735152 ]
Ashish Thusoo commented on HIVE-690:
------------------------------------
That is my bad.
We should just fix this by reading stderr from the spawned process and sending updates to the tracker if something has been written to stderr instead of indefinitely sending updates as that may lead to processes not getting cleaned up?
> ScriptOperator still times out
> ------------------------------
>
> Key: HIVE-690
> URL: https://issues.apache.org/jira/browse/HIVE-690
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-690.1.patch
>
>
> The code is using exp_interval instead of exp_int in the "scheduleAtFxiedRate".
> {code}
> Integer exp_interval = null;
> int exp_int;
> exp_interval = Integer.decode(hconf.get("mapred.tasktracker.expiry.interval"));
> if (exp_interval != null)
> exp_int = exp_interval.intValue() / 2;
> else
> exp_int = 300000;
> rpTimer = new Timer(true);
> rpTimer.scheduleAtFixedRate(new ReporterTask(reporter), 0, exp_interval);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.