You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Gopal V (JIRA)" <ji...@apache.org> on 2015/07/22 23:38:06 UTC
[jira] [Comment Edited] (TEZ-2632) A -Paws and -Pazure build
profiles for hadoop-{aws,azure} inclusion
[ https://issues.apache.org/jira/browse/TEZ-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637681#comment-14637681 ]
Gopal V edited comment on TEZ-2632 at 7/22/15 9:38 PM:
-------------------------------------------------------
bq. s/..., you must explicitly bundle ... for Tez to be functional in the relevant environments/ ?
Actually, this works already with a manual step, this patch is just removing an explicit manual step.
The status quo method is to use tez.aux.uris to ship the S3A jars.
Since Google Storage doesn't have a hadoop-* equivalent, that's how the Google Cloud Storage would work today. By copying the GS HCFS jars to {{/apps/tez/aux-jars/}} and setting up the {{tez.aux.uris}} to point to that.
was (Author: gopalv):
bq. s/..., you must explicitly bundle ... for Tez to be functional in the relevant environments/ ?
Actually, this works already with a manual step, this patch is just removing an explicit manual step.
The status quo method is to use tez.aux.uris to ship the S3A jars.
Since Google Storage doesn't have a hadoop-* equivalent, that's how the Google Cloud Storage would work today. By copying the GS HCFS jars to ${fs.defaultFS}/apps/tez/aux-jars/ and setting up the tez.aux.uris to point to that.
> A -Paws and -Pazure build profiles for hadoop-{aws,azure} inclusion
> -------------------------------------------------------------------
>
> Key: TEZ-2632
> URL: https://issues.apache.org/jira/browse/TEZ-2632
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.6.2, 0.8.0, 0.7.1
> Reporter: Gopal V
> Assignee: Gopal V
> Attachments: TEZ-2632.1.patch, TEZ-2632.2.patch
>
>
> A number of S3a workloads which work with MRv2 (after HADOOP-10400) is failing to work with Tez.
> That compatibility issue can be bypassed with a trivial build profile switch (same for azure after HADOOP-9629)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)