[jira] [Commented] (IMPALA-9293) Impala Doc: Revise explanation of HDFS trashcan usage on S3

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-9293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014673#comment-17014673 ] 

ASF subversion and git services commented on IMPALA-9293:
---------------------------------------------------------

Commit 64828f8b765ad9a8f2cc0f4aede9a1d4def235d7 in impala's branch refs/heads/master from Sahil Takiar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=64828f8 ]

IMPALA-9293: [DOCS] Impala Doc: Revise explanation of HDFS trashcan usage on S3

Updated impala_s3.xml to refer to the trashcan as the "S3A trashcan"
rather than the "HDFS trashcan".

Change-Id: If321117b0d58e3f6d79251fad97b8bd92882cc12
Reviewed-on: http://gerrit.cloudera.org:8080/15022
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Joe McDonnell <jo...@cloudera.com>


> Impala Doc: Revise explanation of HDFS trashcan usage on S3
> -----------------------------------------------------------
>
>                 Key: IMPALA-9293
>                 URL: https://issues.apache.org/jira/browse/IMPALA-9293
>             Project: IMPALA
>          Issue Type: Task
>          Components: Docs
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>            Priority: Major
>
> The Impala docs state:
> {quote}
> By default, when you drop an internal (managed) table, the data files are moved to the HDFS trashcan. This operation is expensive for tables that reside on the Amazon S3 filesystem. Therefore, for S3 tables, prefer to use DROP TABLE table_name PURGE rather than the default DROP TABLE statement. The PURGE clause makes Impala delete the data files immediately, skipping the HDFS trashcan.
> {quote}
> and
> {quote}
> The default DROP TABLE/PARTITION is slow because Impala copies the files to the HDFS trash folder, and Impala waits until all the data is moved. DROP TABLE/PARTITION .. PURGE is a fast delete operation, and the Impala statement finishes quickly even though the change might not have propagated fully throughout S3.
> {quote}
> The confusing part is "Impala copies the files to the HDFS trash folder". Users might think that when a managed Impala table on S3 is dropped, Impala actually copies the data from S3 to a trashcan folder *stored on HDFS*. This isn't true. The term "HDFS trashcan" is used to refer to a feature of HDFS where all deleted data is moved to a trash folder rather than being deleted immediately. See https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html#File+Deletes+and+Undeletes for details.
> What actually happens is that there is a trashcan folder on S3 itself, and when a S3 managed table is dropped, the data is copied from from the managed table folder to the trashcan folder *stored on S3*.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org