You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org> on 2010/09/08 04:12:37 UTC
[jira] Created: (HIVE-1620) Patch to write directly to S3 from Hive
Patch to write directly to S3 from Hive
---------------------------------------
Key: HIVE-1620
URL: https://issues.apache.org/jira/browse/HIVE-1620
Project: Hadoop Hive
Issue Type: New Feature
Reporter: Vaibhav Aggarwal
We want to submit a patch to Hive which allows user to write files directly to S3.
This patch allow user to specify an S3 location as the table output location and hence eliminates the need of copying data from HDFS to S3.
Users can run Hive queries directly over the data stored in S3.
This patch helps integrate hive with S3 better and quicker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1620) Patch to write directly to S3 from
Hive
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12912828#action_12912828 ]
Vaibhav Aggarwal commented on HIVE-1620:
----------------------------------------
I tried to change create a new class S3FileSinkOperator but there seems to be a lot of complexity involved in extending the existing FileSinkOperator class.
Most of the changes that are required are in createBucketFiles() method. Overriding that method will lead to a lot of repeated code which would be very hard to maintain. That method needs to be refactored into smaller methods in order to extend FileSinkOperator. I should be able to do it but that seemed to defeat the purpose of not changing FileSinkOperator much. Please let me know if you are OK with refactoring the FileSinkOperator class into smaller methods.
Based on my investigations I still feel that the current approach is better. You would notice that there are very few changes to the FileSinkOperator in the current patch.
I have just introduced a new variable "fsSupportsMove" which is always parallel to isNativeTable (an existing boolean variable).
The only reason I choose not to reuse isNativeTable variable is to allow the functionality of non-native tables to grow independent of the file systems not supporting move.
Please review the patch one more time considering the above argument and let me know which approach do you think is best.
Thanks
Vaibhav
> Patch to write directly to S3 from Hive
> ---------------------------------------
>
> Key: HIVE-1620
> URL: https://issues.apache.org/jira/browse/HIVE-1620
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1620.patch
>
>
> We want to submit a patch to Hive which allows user to write files directly to S3.
> This patch allow user to specify an S3 location as the table output location and hence eliminates the need of copying data from HDFS to S3.
> Users can run Hive queries directly over the data stored in S3.
> This patch helps integrate hive with S3 better and quicker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1620) Patch to write directly to S3 from
Hive
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920808#action_12920808 ]
Joydeep Sen Sarma commented on HIVE-1620:
-----------------------------------------
i agree that the speed efficiency may be worth the tradeoff in consistency. as you say - the messaging is critical. can we gate this feature on a new hive option that makes the user conscious of this tradeoff?
regarding the cleanup - please look at jobClose method in FileSinkOperator (I think). if the hive client is still functioning at the time the job fails - we can make an attempt to clean things up there (assuming that the file names are unique - which i am not sure about right now because we made some changes to shorten file names (that might have to be undone for this feature)).
one thing we have experienced in the past is that hadoop tasks continue to do stuff even after the job is technically 'complete'. so i think while the cleanup can help the 99% use case - there will be marginal cases where the output directory gets written to when it shouldn't. so having this gated on an option would still be worthwhile IMHO (for users who cannot afford speed-accuracy tradeoff).
> Patch to write directly to S3 from Hive
> ---------------------------------------
>
> Key: HIVE-1620
> URL: https://issues.apache.org/jira/browse/HIVE-1620
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1620.patch
>
>
> We want to submit a patch to Hive which allows user to write files directly to S3.
> This patch allow user to specify an S3 location as the table output location and hence eliminates the need of copying data from HDFS to S3.
> Users can run Hive queries directly over the data stored in S3.
> This patch helps integrate hive with S3 better and quicker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1620) Patch to write directly to S3 from
Hive
Posted by "Richard Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920788#action_12920788 ]
Richard Cole commented on HIVE-1620:
------------------------------------
Hi Joydeep,
The patch as proposed makes no attempt to cleanup the output directory in the case that jobflow failed.
If I write to a table or directory into Amazon S3 and the Hadoop job ultimately fails then the directory contents will have been modified. Previous results will have been removed. Concequently it is not easy to preserve the property that Hive statement failure implies to no change to the destination. Given that the destination may change even though the statement fails, how important do you consider it that the result be no records, rather than a partial set of records?
I know that with HDFS you attempt to achieve atomicity by doing a directory move at the end of the job. However Amazon S3 doesn't have an atomic directory move so this isn't possible. Writing directly to S3 gives a large efficiency gain without making the situation worse than it is today. It is important to message recognise however the different semantics of the two file stores.
If you think it is important to output empty results rather than partial results we can look into that. Where do you think is the best place in Hive to react to the failure of a job for example by cleaning up spurious output from successful task attempts?
regards,
Richard.
> Patch to write directly to S3 from Hive
> ---------------------------------------
>
> Key: HIVE-1620
> URL: https://issues.apache.org/jira/browse/HIVE-1620
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1620.patch
>
>
> We want to submit a patch to Hive which allows user to write files directly to S3.
> This patch allow user to specify an S3 location as the table output location and hence eliminates the need of copying data from HDFS to S3.
> Users can run Hive queries directly over the data stored in S3.
> This patch helps integrate hive with S3 better and quicker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1620) Patch to write directly to S3 from
Hive
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12912849#action_12912849 ]
Namit Jain commented on HIVE-1620:
----------------------------------
I will take a look and get back to you
> Patch to write directly to S3 from Hive
> ---------------------------------------
>
> Key: HIVE-1620
> URL: https://issues.apache.org/jira/browse/HIVE-1620
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1620.patch
>
>
> We want to submit a patch to Hive which allows user to write files directly to S3.
> This patch allow user to specify an S3 location as the table output location and hence eliminates the need of copying data from HDFS to S3.
> Users can run Hive queries directly over the data stored in S3.
> This patch helps integrate hive with S3 better and quicker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1620) Patch to write directly to S3 from
Hive
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910886#action_12910886 ]
Vaibhav Aggarwal commented on HIVE-1620:
----------------------------------------
I think that is a good suggestion.
I will try to break down the functionality into a separate class and will resubmit the patch.
> Patch to write directly to S3 from Hive
> ---------------------------------------
>
> Key: HIVE-1620
> URL: https://issues.apache.org/jira/browse/HIVE-1620
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1620.patch
>
>
> We want to submit a patch to Hive which allows user to write files directly to S3.
> This patch allow user to specify an S3 location as the table output location and hence eliminates the need of copying data from HDFS to S3.
> Users can run Hive queries directly over the data stored in S3.
> This patch helps integrate hive with S3 better and quicker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1620) Patch to write directly to S3 from
Hive
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915383#action_12915383 ]
Vaibhav Aggarwal commented on HIVE-1620:
----------------------------------------
Hey Namit
Please let me know what you think of this.
Thanks
Vaibhav
> Patch to write directly to S3 from Hive
> ---------------------------------------
>
> Key: HIVE-1620
> URL: https://issues.apache.org/jira/browse/HIVE-1620
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1620.patch
>
>
> We want to submit a patch to Hive which allows user to write files directly to S3.
> This patch allow user to specify an S3 location as the table output location and hence eliminates the need of copying data from HDFS to S3.
> Users can run Hive queries directly over the data stored in S3.
> This patch helps integrate hive with S3 better and quicker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1620) Patch to write directly to S3 from
Hive
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918611#action_12918611 ]
Joydeep Sen Sarma commented on HIVE-1620:
-----------------------------------------
if we are overwriting a table ('insert overwrite table') - do we make sure that if the query/job fails in between - then some of the files (from maps/reduces that did succeed) are not left in the table's directory?
> Patch to write directly to S3 from Hive
> ---------------------------------------
>
> Key: HIVE-1620
> URL: https://issues.apache.org/jira/browse/HIVE-1620
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1620.patch
>
>
> We want to submit a patch to Hive which allows user to write files directly to S3.
> This patch allow user to specify an S3 location as the table output location and hence eliminates the need of copying data from HDFS to S3.
> Users can run Hive queries directly over the data stored in S3.
> This patch helps integrate hive with S3 better and quicker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1620) Patch to write directly to S3 from
Hive
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907349#action_12907349 ]
Vaibhav Aggarwal commented on HIVE-1620:
----------------------------------------
This is the reason the patch uses task id instead of attempt id to write to s3.
Each process writes to the same file. In case of s3 the last process to commit the file wins.
Hadoop tasks are supposed to be idempotent hence this should work.
> Patch to write directly to S3 from Hive
> ---------------------------------------
>
> Key: HIVE-1620
> URL: https://issues.apache.org/jira/browse/HIVE-1620
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1620.patch
>
>
> We want to submit a patch to Hive which allows user to write files directly to S3.
> This patch allow user to specify an S3 location as the table output location and hence eliminates the need of copying data from HDFS to S3.
> Users can run Hive queries directly over the data stored in S3.
> This patch helps integrate hive with S3 better and quicker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1620) Patch to write directly to S3 from
Hive
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907093#action_12907093 ]
Namit Jain commented on HIVE-1620:
----------------------------------
How will this handle failures and speculative execution ?
If you are directly writing to a file, cant it lead to duplicate data ?
> Patch to write directly to S3 from Hive
> ---------------------------------------
>
> Key: HIVE-1620
> URL: https://issues.apache.org/jira/browse/HIVE-1620
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1620.patch
>
>
> We want to submit a patch to Hive which allows user to write files directly to S3.
> This patch allow user to specify an S3 location as the table output location and hence eliminates the need of copying data from HDFS to S3.
> Users can run Hive queries directly over the data stored in S3.
> This patch helps integrate hive with S3 better and quicker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1620) Patch to write directly to S3 from
Hive
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain reassigned HIVE-1620:
--------------------------------
Assignee: Vaibhav Aggarwal
> Patch to write directly to S3 from Hive
> ---------------------------------------
>
> Key: HIVE-1620
> URL: https://issues.apache.org/jira/browse/HIVE-1620
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1620.patch
>
>
> We want to submit a patch to Hive which allows user to write files directly to S3.
> This patch allow user to specify an S3 location as the table output location and hence eliminates the need of copying data from HDFS to S3.
> Users can run Hive queries directly over the data stored in S3.
> This patch helps integrate hive with S3 better and quicker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1620) Patch to write directly to S3 from Hive
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vaibhav Aggarwal updated HIVE-1620:
-----------------------------------
Attachment: HIVE-1620.patch
> Patch to write directly to S3 from Hive
> ---------------------------------------
>
> Key: HIVE-1620
> URL: https://issues.apache.org/jira/browse/HIVE-1620
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Vaibhav Aggarwal
> Attachments: HIVE-1620.patch
>
>
> We want to submit a patch to Hive which allows user to write files directly to S3.
> This patch allow user to specify an S3 location as the table output location and hence eliminates the need of copying data from HDFS to S3.
> Users can run Hive queries directly over the data stored in S3.
> This patch helps integrate hive with S3 better and quicker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1620) Patch to write directly to S3 from
Hive
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908854#action_12908854 ]
Namit Jain commented on HIVE-1620:
----------------------------------
The approach looks good, but can you move all the checks to compile time instead.
I mean, while generating a plan, create a S3FileSinkOperator instead of FileSinkOperator, if the
destination under consideration is on S3 FileSystem - there will be no move task etc.
The explain work will show the correct plan
The commit for S3FileSystem will be a no-op. That way, FileSinkOperator does not change much
> Patch to write directly to S3 from Hive
> ---------------------------------------
>
> Key: HIVE-1620
> URL: https://issues.apache.org/jira/browse/HIVE-1620
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1620.patch
>
>
> We want to submit a patch to Hive which allows user to write files directly to S3.
> This patch allow user to specify an S3 location as the table output location and hence eliminates the need of copying data from HDFS to S3.
> Users can run Hive queries directly over the data stored in S3.
> This patch helps integrate hive with S3 better and quicker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.