You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "S. Alex Smith (JIRA)" <ji...@apache.org> on 2009/08/26 03:29:59 UTC
[jira] Created: (HIVE-797) mappers should report life in ways other
than emitting data
mappers should report life in ways other than emitting data
-----------------------------------------------------------
Key: HIVE-797
URL: https://issues.apache.org/jira/browse/HIVE-797
Project: Hadoop Hive
Issue Type: Bug
Reporter: S. Alex Smith
Mappers which are performing a great deal of aggregation can be killed by time out even if they are running successfully. For example, in the following query the group by operator stops the mapper from returning any rows of data until the map is entirely finished. If the data processing takes longer than the time-out limit, the job will fail. The mapper should instead offer the tracker some indication that it is busy working. Alternatively, the tracker could ping the mapper with an appropriate question / warning before it sends a kill signal.
FROM (
FROM my_table
SELECT TRANSFORM(my_data)
USING 'my_boolean_function'
AS boolean_output) a
SELECT boolean_output, COUNT(1)
GROUP BY boolean_output
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-797) mappers should report life in ways
other than emitting data
Posted by "S. Alex Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747840#action_12747840 ]
S. Alex Smith commented on HIVE-797:
------------------------------------
setting
set hive.script.auto.progress=true;
seems to have no effect. Aside from the job succeeding (it doesn't), what effect should I be able to measure (in order to see if this is doing anything)?
> mappers should report life in ways other than emitting data
> -----------------------------------------------------------
>
> Key: HIVE-797
> URL: https://issues.apache.org/jira/browse/HIVE-797
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: S. Alex Smith
>
> Mappers which are performing a great deal of aggregation can be killed by time out even if they are running successfully. For example, in the following query the group by operator stops the mapper from returning any rows of data until the map is entirely finished. If the data processing takes longer than the time-out limit, the job will fail. The mapper should instead offer the tracker some indication that it is busy working. Alternatively, the tracker could ping the mapper with an appropriate question / warning before it sends a kill signal.
> FROM (
> FROM my_table
> SELECT TRANSFORM(my_data)
> USING 'my_boolean_function'
> AS boolean_output) a
> SELECT boolean_output, COUNT(1)
> GROUP BY boolean_output
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-797) mappers should report life in ways
other than emitting data
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao resolved HIVE-797.
-----------------------------
Resolution: Fixed
> mappers should report life in ways other than emitting data
> -----------------------------------------------------------
>
> Key: HIVE-797
> URL: https://issues.apache.org/jira/browse/HIVE-797
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: S. Alex Smith
>
> Mappers which are performing a great deal of aggregation can be killed by time out even if they are running successfully. For example, in the following query the group by operator stops the mapper from returning any rows of data until the map is entirely finished. If the data processing takes longer than the time-out limit, the job will fail. The mapper should instead offer the tracker some indication that it is busy working. Alternatively, the tracker could ping the mapper with an appropriate question / warning before it sends a kill signal.
> FROM (
> FROM my_table
> SELECT TRANSFORM(my_data)
> USING 'my_boolean_function'
> AS boolean_output) a
> SELECT boolean_output, COUNT(1)
> GROUP BY boolean_output
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-797) mappers should report life in ways
other than emitting data
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747776#action_12747776 ]
Zheng Shao commented on HIVE-797:
---------------------------------
Mapper script can write to stderr to avoid the killing.
Alternatively, you can do the following to achieve the same result:
{code}
set hive.script.auto.progress=true;
{code}
> mappers should report life in ways other than emitting data
> -----------------------------------------------------------
>
> Key: HIVE-797
> URL: https://issues.apache.org/jira/browse/HIVE-797
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: S. Alex Smith
>
> Mappers which are performing a great deal of aggregation can be killed by time out even if they are running successfully. For example, in the following query the group by operator stops the mapper from returning any rows of data until the map is entirely finished. If the data processing takes longer than the time-out limit, the job will fail. The mapper should instead offer the tracker some indication that it is busy working. Alternatively, the tracker could ping the mapper with an appropriate question / warning before it sends a kill signal.
> FROM (
> FROM my_table
> SELECT TRANSFORM(my_data)
> USING 'my_boolean_function'
> AS boolean_output) a
> SELECT boolean_output, COUNT(1)
> GROUP BY boolean_output
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.