[jira] [Commented] (TAJO-1211) Staging directory for CTAS and INSERT should be in the output dir.

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TAJO-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14229401#comment-14229401 ] 

ASF GitHub Bot commented on TAJO-1211:
--------------------------------------

Github user hyunsik commented on the pull request:

    https://github.com/apache/tajo/pull/274#issuecomment-65019661
  
    I updated the patch. This patch also fixes a bug that misses exceptions occurring in Query::commitOuputData.


> Staging directory for CTAS and INSERT should be in the output dir.
> ------------------------------------------------------------------
>
>                 Key: TAJO-1211
>                 URL: https://issues.apache.org/jira/browse/TAJO-1211
>             Project: Tajo
>          Issue Type: Bug
>          Components: query master
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
>             Fix For: 0.9.1
>
>
> *Background*
> Staging directory plays a role to keep the final output data temporarily. The final output data are moved toe the the final output dir if query is successfully finished. It is important to keep the output directory consistent even if query is failed.
> *Problem*
> Currently, staging directory is included /tmp/tajo-$\{user.name\}/ in HDFS that $\{tajo.root\} uses. The final output directory and the staging directory can be on different file systems. In this case, the move will cause unnecessary copy overheads. In addition, in S3, such a move operation may be more problematic.
> *Solution*
> CTAS and INSERT (OVERWRITE) INTO should use the staging dir as a hidden subdirectory in the final output dir. For example, if the output dir is {{/table1}}, the corresponding staging dir should be {{/table1/.staging}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)