You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tez.apache.org by Chris K Wensel <ch...@wensel.net> on 2015/02/24 23:34:47 UTC
tez yarn timeline history maven artifacts
As of Tez 0.6.0 I hear the intent is to support Hadoop 2.4 -> 2.6, and that 2.4/2.5 don’t have a secure yarn timeline server.
I see there is a tez-yarn-timeline-history-with-acls artifact in maven for 0.6.0, which I assume is for a Hadoop 2.6+ deployment. This is also reflected in the poms for the 0.6.0 tez release.
I see there is not a tez-yarn-timeline-history artifact in maven that would, I believe, provide backwards compatibility with 2.4. though the pom does list its in an alternative build profile.
https://repository.apache.org/content/repositories/releases/org/apache/tez/tez-yarn-timeline-history/ <https://repository.apache.org/content/repositories/releases/org/apache/tez/tez-yarn-timeline-history/>
Is not building and publishing this artifact by design, or an oversight?
Or will the tez-yarn-timeline-history-with-acls artifact work with Hadoop 2.4/2.5?
ckw
—
Chris K Wensel
chris@wensel.net
Re: tez yarn timeline history maven artifacts
Posted by Chris K Wensel <ch...@wensel.net>.
Cascading needs these dependencies as a compile dependency, at runtime they are optional.
I'm simply looking for them to exist so that i can automate tests against 2.4 and 2.6. I have no issues handling conditional dependencies (this is a requirement since every Hadoop release leaves stuff out anyway)
https://github.com/cwensel/cascading/blob/wip-3.0/cascading-hadoop2-tez/build.gradle#L66
We take great pains with retaining compatibility:
http://www.cascading.org/support/compatibility/
Also the missing deployable artifacts are a pain as well. I am already providing them to the Cascading community for testing purposes.
https://github.com/cwensel/cascading/tree/wip-3.0/cascading-hadoop2-tez#running-a-tez-application
As an aside, please feel free to update the README to reflect Cascading's ongoing support for Tez
https://github.com/apache/tez/blob/master/README.md
ckw
> On Feb 25, 2015, at 11:45 AM, Hitesh Shah <hi...@apache.org> wrote:
>
> Hello Chris
>
> From a supported point of view, both 2.4 and and 2.6 are supported. However, ACLs support was not built into Timeline in Hadoop 2.4 hence a fully secure setup cannot be supported with 2.4. Some of this is explained in http://tez.apache.org/tez_yarn_timeline.html.
> For the uploaded artifacts, not publishing the tez-yarn-timeline-history artifact is something which I am aware of but I ignored it to some extent as it does not contain any APIs that a user would need to compile against. Just to be clear, the tez-yarn-timeline-history-with-acls artifact will not work with 2.4 as it needs specific APIs introduced in hadoop 2.6.
>
> From a deployment of view, a few folks have called for the need of publishing binary artifacts for convenience so that it makes it easy for users to deploy Tez. In this regard, we ( the Tez community ) will probably need to provide multiple sets compiled against different versions of hadoop. There are a couple of reasons for this. The main one is obviously Timeline and what features it supports or does not support in a particular version of Hadoop ( and the fact that it did not exist in 2.2/2.3 ). The other is that YARN did introduce some incompatibilities between 2.4 and 2.6 hence there are some edge cases where an app compiled against 2.4 will not work on a 2.6 cluster.
>
> That said, are you looking to use any particular functionality from within the tez-yarn-timeline-history-with-acls jar or looking to be able download the Tez artifacts that one can deploy for use with a particular version of hadoop?
>
> thanks
> — Hitesh
>
>
> On Feb 24, 2015, at 2:34 PM, Chris K Wensel <ch...@wensel.net> wrote:
>
>>
>> As of Tez 0.6.0 I hear the intent is to support Hadoop 2.4 -> 2.6, and that 2.4/2.5 don’t have a secure yarn timeline server.
>>
>> I see there is a tez-yarn-timeline-history-with-acls artifact in maven for 0.6.0, which I assume is for a Hadoop 2.6+ deployment. This is also reflected in the poms for the 0.6.0 tez release.
>>
>> I see there is not a tez-yarn-timeline-history artifact in maven that would, I believe, provide backwards compatibility with 2.4. though the pom does list its in an alternative build profile.
>>
>> https://repository.apache.org/content/repositories/releases/org/apache/tez/tez-yarn-timeline-history/ <https://repository.apache.org/content/repositories/releases/org/apache/tez/tez-yarn-timeline-history/>
>>
>> Is not building and publishing this artifact by design, or an oversight?
>>
>> Or will the tez-yarn-timeline-history-with-acls artifact work with Hadoop 2.4/2.5?
>>
>> ckw
>>
>> —
>> Chris K Wensel
>> chris@wensel.net
>>
>>
>>
>>
>
—
Chris K Wensel
chris@wensel.net
Re: tez yarn timeline history maven artifacts
Posted by Hitesh Shah <hi...@apache.org>.
Hello Chris
From a supported point of view, both 2.4 and and 2.6 are supported. However, ACLs support was not built into Timeline in Hadoop 2.4 hence a fully secure setup cannot be supported with 2.4. Some of this is explained in http://tez.apache.org/tez_yarn_timeline.html.
For the uploaded artifacts, not publishing the tez-yarn-timeline-history artifact is something which I am aware of but I ignored it to some extent as it does not contain any APIs that a user would need to compile against. Just to be clear, the tez-yarn-timeline-history-with-acls artifact will not work with 2.4 as it needs specific APIs introduced in hadoop 2.6.
From a deployment of view, a few folks have called for the need of publishing binary artifacts for convenience so that it makes it easy for users to deploy Tez. In this regard, we ( the Tez community ) will probably need to provide multiple sets compiled against different versions of hadoop. There are a couple of reasons for this. The main one is obviously Timeline and what features it supports or does not support in a particular version of Hadoop ( and the fact that it did not exist in 2.2/2.3 ). The other is that YARN did introduce some incompatibilities between 2.4 and 2.6 hence there are some edge cases where an app compiled against 2.4 will not work on a 2.6 cluster.
That said, are you looking to use any particular functionality from within the tez-yarn-timeline-history-with-acls jar or looking to be able download the Tez artifacts that one can deploy for use with a particular version of hadoop?
thanks
— Hitesh
On Feb 24, 2015, at 2:34 PM, Chris K Wensel <ch...@wensel.net> wrote:
>
> As of Tez 0.6.0 I hear the intent is to support Hadoop 2.4 -> 2.6, and that 2.4/2.5 don’t have a secure yarn timeline server.
>
> I see there is a tez-yarn-timeline-history-with-acls artifact in maven for 0.6.0, which I assume is for a Hadoop 2.6+ deployment. This is also reflected in the poms for the 0.6.0 tez release.
>
> I see there is not a tez-yarn-timeline-history artifact in maven that would, I believe, provide backwards compatibility with 2.4. though the pom does list its in an alternative build profile.
>
> https://repository.apache.org/content/repositories/releases/org/apache/tez/tez-yarn-timeline-history/ <https://repository.apache.org/content/repositories/releases/org/apache/tez/tez-yarn-timeline-history/>
>
> Is not building and publishing this artifact by design, or an oversight?
>
> Or will the tez-yarn-timeline-history-with-acls artifact work with Hadoop 2.4/2.5?
>
> ckw
>
> —
> Chris K Wensel
> chris@wensel.net
>
>
>
>