[jira] [Updated] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution

["Zhiyuan Yang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhiyuan Yang updated HIVE-17641:
--------------------------------
    Attachment: HIVE-17641.2-branch-0.14.patch

Thanks [~ashutoshc] for review! I've attached new patch to address comment. Actually that exception msg can be omit because other place already print how many task in total, how many task has been executed. LOG.info won't be noise. That function is not called a lot. Anyway, fixed all of them in new patch.

> Visibility issue of Task.done cause Driver skip stages in parallel execution
> ----------------------------------------------------------------------------
>
>                 Key: HIVE-17641
>                 URL: https://issues.apache.org/jira/browse/HIVE-17641
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.14.1
>            Reporter: Zhiyuan Yang
>            Assignee: Zhiyuan Yang
>         Attachments: HIVE-17641.1-branch-0.14.patch, HIVE-17641.2-branch-0.14.patch
>
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread set this value, and Driver thread read this value when it determines whether a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task<? extends Serializable> tsk) {
>     return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
>     boolean isrunnable = true;
>     if (parentTasks != null) {
>       for (Task<? extends Serializable> parent : parentTasks) {
>         if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even all parents finish.
> To make it worse, Driver think query is successful when there is no running task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
>     return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)