You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2012/09/18 13:24:08 UTC
[jira] [Created] (HIVE-3477) Duplicate data possible with
speculative execution for dynamic partitions
Namit Jain created HIVE-3477:
--------------------------------
Summary: Duplicate data possible with speculative execution for dynamic partitions
Key: HIVE-3477
URL: https://issues.apache.org/jira/browse/HIVE-3477
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Namit Jain
Assignee: Namit Jain
Consider a query like:
insert overwrite T partition (ds)
select * from
(mapreduce-subq1
union all
mapreduce-subq2)x;
Once, mapreduce-subq1 and mapreduce-subq2 are done, the task for the union
is invoked. At the end of the union task, jobClose is invoked.
Note that there are 2 tablescan operators. The tree is something like:
TABLESCAN1 --
\
UNION -- SELECT -- FILESINK
/
TABLESCAN2 --
In the current setup, jobClose will be invoked twice for FileSink.
In case of speculative execution, it is possible that data is still is
being written to tmp Dir. after jobClose is finished once.
The correct fix would be to make sure that jobClose is only invoked once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3477) Duplicate data possible with
speculative execution for dynamic partitions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-3477:
-----------------------------
Attachment: hive.3477.1.patch
> Duplicate data possible with speculative execution for dynamic partitions
> -------------------------------------------------------------------------
>
> Key: HIVE-3477
> URL: https://issues.apache.org/jira/browse/HIVE-3477
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.3477.1.patch
>
>
> Consider a query like:
> insert overwrite T partition (ds)
> select * from
> (mapreduce-subq1
> union all
> mapreduce-subq2)x;
> Once, mapreduce-subq1 and mapreduce-subq2 are done, the task for the union
> is invoked. At the end of the union task, jobClose is invoked.
> Note that there are 2 tablescan operators. The tree is something like:
> TABLESCAN1 --
> \
> UNION -- SELECT -- FILESINK
> /
> TABLESCAN2 --
> In the current setup, jobClose will be invoked twice for FileSink.
> In case of speculative execution, it is possible that data is still is
> being written to tmp Dir. after jobClose is finished once.
> The correct fix would be to make sure that jobClose is only invoked once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3477) Duplicate data possible with
speculative execution for dynamic partitions
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458980#comment-13458980 ]
Ashutosh Chauhan commented on HIVE-3477:
----------------------------------------
Cool. I created HIVE-3489
> Duplicate data possible with speculative execution for dynamic partitions
> -------------------------------------------------------------------------
>
> Key: HIVE-3477
> URL: https://issues.apache.org/jira/browse/HIVE-3477
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.3477.1.patch
>
>
> Consider a query like:
> insert overwrite T partition (ds)
> select * from
> (mapreduce-subq1
> union all
> mapreduce-subq2)x;
> Once, mapreduce-subq1 and mapreduce-subq2 are done, the task for the union
> is invoked. At the end of the union task, jobClose is invoked.
> Note that there are 2 tablescan operators. The tree is something like:
> TABLESCAN1 --
> \
> UNION -- SELECT -- FILESINK
> /
> TABLESCAN2 --
> In the current setup, jobClose will be invoked twice for FileSink.
> In case of speculative execution, it is possible that data is still is
> being written to tmp Dir. after jobClose is finished once.
> The correct fix would be to make sure that jobClose is only invoked once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3477) Duplicate data possible with
speculative execution for dynamic partitions
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458017#comment-13458017 ]
Ashutosh Chauhan commented on HIVE-3477:
----------------------------------------
In general I think we should start adopting {{OutputCommitter}} functionality provided by MR framework instead of having our own {{JobClose}} operator. Whatever logic we want to run at the end of the query (which is done via JobClose currently) can be run in {{commitJob}} of last MR job for the query. This insulates us from having to keep a live hive client around for long running queries. Secondly, we are replicating functionality which is already provided by underlying framework. Possibly, this may also help us in avoiding bugs like this one.
This definitely is not in scope of this bug, but wanted to bring this up since while fixing this issue you might have more insights into why or why not this may be a good idea.
> Duplicate data possible with speculative execution for dynamic partitions
> -------------------------------------------------------------------------
>
> Key: HIVE-3477
> URL: https://issues.apache.org/jira/browse/HIVE-3477
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.3477.1.patch
>
>
> Consider a query like:
> insert overwrite T partition (ds)
> select * from
> (mapreduce-subq1
> union all
> mapreduce-subq2)x;
> Once, mapreduce-subq1 and mapreduce-subq2 are done, the task for the union
> is invoked. At the end of the union task, jobClose is invoked.
> Note that there are 2 tablescan operators. The tree is something like:
> TABLESCAN1 --
> \
> UNION -- SELECT -- FILESINK
> /
> TABLESCAN2 --
> In the current setup, jobClose will be invoked twice for FileSink.
> In case of speculative execution, it is possible that data is still is
> being written to tmp Dir. after jobClose is finished once.
> The correct fix would be to make sure that jobClose is only invoked once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3477) Duplicate data possible with
speculative execution for dynamic partitions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458844#comment-13458844 ]
Namit Jain commented on HIVE-3477:
----------------------------------
All the tests passed.
[~ashutoshc], we can can definitely discuss it in a new jira.
We dont use OutputCommitter at all right now - since we write into tmp. directories and then move the temp. dirs into the
final dir.
> Duplicate data possible with speculative execution for dynamic partitions
> -------------------------------------------------------------------------
>
> Key: HIVE-3477
> URL: https://issues.apache.org/jira/browse/HIVE-3477
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.3477.1.patch
>
>
> Consider a query like:
> insert overwrite T partition (ds)
> select * from
> (mapreduce-subq1
> union all
> mapreduce-subq2)x;
> Once, mapreduce-subq1 and mapreduce-subq2 are done, the task for the union
> is invoked. At the end of the union task, jobClose is invoked.
> Note that there are 2 tablescan operators. The tree is something like:
> TABLESCAN1 --
> \
> UNION -- SELECT -- FILESINK
> /
> TABLESCAN2 --
> In the current setup, jobClose will be invoked twice for FileSink.
> In case of speculative execution, it is possible that data is still is
> being written to tmp Dir. after jobClose is finished once.
> The correct fix would be to make sure that jobClose is only invoked once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3477) Duplicate data possible with
speculative execution for dynamic partitions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457750#comment-13457750 ]
Namit Jain commented on HIVE-3477:
----------------------------------
https://reviews.facebook.net/D5505
> Duplicate data possible with speculative execution for dynamic partitions
> -------------------------------------------------------------------------
>
> Key: HIVE-3477
> URL: https://issues.apache.org/jira/browse/HIVE-3477
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
>
> Consider a query like:
> insert overwrite T partition (ds)
> select * from
> (mapreduce-subq1
> union all
> mapreduce-subq2)x;
> Once, mapreduce-subq1 and mapreduce-subq2 are done, the task for the union
> is invoked. At the end of the union task, jobClose is invoked.
> Note that there are 2 tablescan operators. The tree is something like:
> TABLESCAN1 --
> \
> UNION -- SELECT -- FILESINK
> /
> TABLESCAN2 --
> In the current setup, jobClose will be invoked twice for FileSink.
> In case of speculative execution, it is possible that data is still is
> being written to tmp Dir. after jobClose is finished once.
> The correct fix would be to make sure that jobClose is only invoked once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3477) Duplicate data possible with
speculative execution for dynamic partitions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-3477:
-----------------------------
Status: Patch Available (was: Open)
> Duplicate data possible with speculative execution for dynamic partitions
> -------------------------------------------------------------------------
>
> Key: HIVE-3477
> URL: https://issues.apache.org/jira/browse/HIVE-3477
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.3477.1.patch
>
>
> Consider a query like:
> insert overwrite T partition (ds)
> select * from
> (mapreduce-subq1
> union all
> mapreduce-subq2)x;
> Once, mapreduce-subq1 and mapreduce-subq2 are done, the task for the union
> is invoked. At the end of the union task, jobClose is invoked.
> Note that there are 2 tablescan operators. The tree is something like:
> TABLESCAN1 --
> \
> UNION -- SELECT -- FILESINK
> /
> TABLESCAN2 --
> In the current setup, jobClose will be invoked twice for FileSink.
> In case of speculative execution, it is possible that data is still is
> being written to tmp Dir. after jobClose is finished once.
> The correct fix would be to make sure that jobClose is only invoked once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3477) Duplicate data possible with
speculative execution for dynamic partitions
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-3477:
--------------------------------
Resolution: Fixed
Fix Version/s: 0.10.0
Status: Resolved (was: Patch Available)
Committed, thanks Namit.
> Duplicate data possible with speculative execution for dynamic partitions
> -------------------------------------------------------------------------
>
> Key: HIVE-3477
> URL: https://issues.apache.org/jira/browse/HIVE-3477
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Fix For: 0.10.0
>
> Attachments: hive.3477.1.patch
>
>
> Consider a query like:
> insert overwrite T partition (ds)
> select * from
> (mapreduce-subq1
> union all
> mapreduce-subq2)x;
> Once, mapreduce-subq1 and mapreduce-subq2 are done, the task for the union
> is invoked. At the end of the union task, jobClose is invoked.
> Note that there are 2 tablescan operators. The tree is something like:
> TABLESCAN1 --
> \
> UNION -- SELECT -- FILESINK
> /
> TABLESCAN2 --
> In the current setup, jobClose will be invoked twice for FileSink.
> In case of speculative execution, it is possible that data is still is
> being written to tmp Dir. after jobClose is finished once.
> The correct fix would be to make sure that jobClose is only invoked once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3477) Duplicate data possible with
speculative execution for dynamic partitions
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457933#comment-13457933 ]
Kevin Wilfong commented on HIVE-3477:
-------------------------------------
+1 Nice catch Namit.
> Duplicate data possible with speculative execution for dynamic partitions
> -------------------------------------------------------------------------
>
> Key: HIVE-3477
> URL: https://issues.apache.org/jira/browse/HIVE-3477
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.3477.1.patch
>
>
> Consider a query like:
> insert overwrite T partition (ds)
> select * from
> (mapreduce-subq1
> union all
> mapreduce-subq2)x;
> Once, mapreduce-subq1 and mapreduce-subq2 are done, the task for the union
> is invoked. At the end of the union task, jobClose is invoked.
> Note that there are 2 tablescan operators. The tree is something like:
> TABLESCAN1 --
> \
> UNION -- SELECT -- FILESINK
> /
> TABLESCAN2 --
> In the current setup, jobClose will be invoked twice for FileSink.
> In case of speculative execution, it is possible that data is still is
> being written to tmp Dir. after jobClose is finished once.
> The correct fix would be to make sure that jobClose is only invoked once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira