You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tez.apache.org by Hitesh Shah <hi...@apache.org> on 2014/12/04 20:36:41 UTC
[DISCUSS] What versions of hadoop should we support?
Hello folks
Could folks who are following these mailing lists do a raise of hands on which versions of Hadoop you are trying to run Tez on?
1) 2.2.x
2) 2.3.x
3) 2.4.x
4) 2.5.x
5) 2.6.x
Given that we are building out the UI and for the large part, the History UI ( i.e. the one which will display DAG details after an app has disappeared ) will be dependent on YARN Timeline, the UI will not be supported for Hadoop versions 2.2.x and 2.3.x given that they do not support Timeline.
Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x - this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
Also, as part of TEZ-1696, I also propose changing the default version to 2.6.0 given that secure Timeline is only available as of 2.6.0.
Thoughts/concerns?
thanks
— Hitesh
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Hitesh Shah <hi...@apache.org>.
Thanks for the useful information on your usage scenarios, folks.
As part of TEZ-1696, to get a fully-secure UI in place, I will go ahead and update the default dependency to hadoop 2.6 ( given that earlier versions currently do not have full secure support ). I will see if we can initiate a discussion in the Hadoop community to see if some of the missing Timeline functionality can be back ported to the hadoop 2.4.x line ( though this will mean that folks on hadoop 2.4.0 will need to be upgrade in any case ).
As for hadoop 2.2.x to 2.5.x, for the most part, all of Tez will continue to work as is with some missing features depending on what features that particular version of hadoop supports. The UI should also be functional against hadoop-2.4 for the most part except where security is enabled.
Details on how to build against a particular version of hadoop will be covered in the BUILDING.txt. For details on Tez and working with various hadoop versions for YARN Timeline, details are at: http://tez.apache.org/tez_yarn_timeline.html
And yes, please raise any issues that you see with the versions of hadoop you are running after TEZ-1696 goes in ( sometime in the next hour or so ).
thanks
— Hitesh
On Dec 11, 2014, at 8:02 AM, Jonathan Eagles <je...@gmail.com> wrote:
> Adding to the discussion. I am using some 2.5 and 2.6
>
> On Wed, Dec 10, 2014 at 8:02 PM, Cheolsoo Park <pi...@gmail.com> wrote:
>> We're using EMR 2.4 in production, and it will take a while to upgrade to
>> 2.6 depending on how fast EMR can provide us 2.6 AMI. But if that's what we
>> need to fully use Tez-8, we will upgrade our cluster.
>>
>> On Wed, Dec 10, 2014 at 1:44 PM, Siddharth Seth <ss...@apache.org> wrote:
>>>
>>> +1 for changing the default to 2.6.0.
>>>
>>> Publishing multiple jars - do you mean per hadoop version supported ?
>>> That'll likely be very difficult to use - what would the dependency be ?
>>> Will there be binary compatibility issues if someone builds against tez jars
>>> built against Hadoop-2.6, and then deployed against 2.2 ?
>>>
>>> On Mon, Dec 8, 2014 at 10:02 AM, Hitesh Shah <hi...@apache.org> wrote:
>>>>
>>>> Thanks for the info, Douglas. We will not be dropping support for 2.4 for
>>>> sure at the moment.
>>>>
>>>> Anyone else have any comments in this regard?
>>>>
>>>> @Bikas/@Sid - what do we do about publishing jars for a release? We would
>>>> need to publish 2 or 3 sets?
>>>>
>>>> thanks
>>>> — Hitesh
>>>>
>>>> On Dec 4, 2014, at 12:51 PM, Moore, Douglas
>>>> <Do...@thinkbiganalytics.com> wrote:
>>>>
>>>>> 2.4.x and beyond for the customers we have
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>>>>>>
>>>>>> Hello folks
>>>>>>
>>>>>> Could folks who are following these mailing lists do a raise of hands
>>>>>> on which versions of Hadoop you are trying to run Tez on?
>>>>>>
>>>>>> 1) 2.2.x
>>>>>> 2) 2.3.x
>>>>>> 3) 2.4.x
>>>>>> 4) 2.5.x
>>>>>> 5) 2.6.x
>>>>>>
>>>>>> Given that we are building out the UI and for the large part, the
>>>>>> History UI ( i.e. the one which will display DAG details after an app has
>>>>>> disappeared ) will be dependent on YARN Timeline, the UI will not be
>>>>>> supported for Hadoop versions 2.2.x and 2.3.x given that they do not support
>>>>>> Timeline.
>>>>>> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x -
>>>>>> this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>>>>>>
>>>>>> Also, as part of TEZ-1696, I also propose changing the default version
>>>>>> to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>>>>>>
>>>>>> Thoughts/concerns?
>>>>>>
>>>>>> thanks
>>>>>> — Hitesh
>>>>>>
>>>>
>>>
>>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Hitesh Shah <hi...@apache.org>.
Thanks for the useful information on your usage scenarios, folks.
As part of TEZ-1696, to get a fully-secure UI in place, I will go ahead and update the default dependency to hadoop 2.6 ( given that earlier versions currently do not have full secure support ). I will see if we can initiate a discussion in the Hadoop community to see if some of the missing Timeline functionality can be back ported to the hadoop 2.4.x line ( though this will mean that folks on hadoop 2.4.0 will need to be upgrade in any case ).
As for hadoop 2.2.x to 2.5.x, for the most part, all of Tez will continue to work as is with some missing features depending on what features that particular version of hadoop supports. The UI should also be functional against hadoop-2.4 for the most part except where security is enabled.
Details on how to build against a particular version of hadoop will be covered in the BUILDING.txt. For details on Tez and working with various hadoop versions for YARN Timeline, details are at: http://tez.apache.org/tez_yarn_timeline.html
And yes, please raise any issues that you see with the versions of hadoop you are running after TEZ-1696 goes in ( sometime in the next hour or so ).
thanks
— Hitesh
On Dec 11, 2014, at 8:02 AM, Jonathan Eagles <je...@gmail.com> wrote:
> Adding to the discussion. I am using some 2.5 and 2.6
>
> On Wed, Dec 10, 2014 at 8:02 PM, Cheolsoo Park <pi...@gmail.com> wrote:
>> We're using EMR 2.4 in production, and it will take a while to upgrade to
>> 2.6 depending on how fast EMR can provide us 2.6 AMI. But if that's what we
>> need to fully use Tez-8, we will upgrade our cluster.
>>
>> On Wed, Dec 10, 2014 at 1:44 PM, Siddharth Seth <ss...@apache.org> wrote:
>>>
>>> +1 for changing the default to 2.6.0.
>>>
>>> Publishing multiple jars - do you mean per hadoop version supported ?
>>> That'll likely be very difficult to use - what would the dependency be ?
>>> Will there be binary compatibility issues if someone builds against tez jars
>>> built against Hadoop-2.6, and then deployed against 2.2 ?
>>>
>>> On Mon, Dec 8, 2014 at 10:02 AM, Hitesh Shah <hi...@apache.org> wrote:
>>>>
>>>> Thanks for the info, Douglas. We will not be dropping support for 2.4 for
>>>> sure at the moment.
>>>>
>>>> Anyone else have any comments in this regard?
>>>>
>>>> @Bikas/@Sid - what do we do about publishing jars for a release? We would
>>>> need to publish 2 or 3 sets?
>>>>
>>>> thanks
>>>> — Hitesh
>>>>
>>>> On Dec 4, 2014, at 12:51 PM, Moore, Douglas
>>>> <Do...@thinkbiganalytics.com> wrote:
>>>>
>>>>> 2.4.x and beyond for the customers we have
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>>>>>>
>>>>>> Hello folks
>>>>>>
>>>>>> Could folks who are following these mailing lists do a raise of hands
>>>>>> on which versions of Hadoop you are trying to run Tez on?
>>>>>>
>>>>>> 1) 2.2.x
>>>>>> 2) 2.3.x
>>>>>> 3) 2.4.x
>>>>>> 4) 2.5.x
>>>>>> 5) 2.6.x
>>>>>>
>>>>>> Given that we are building out the UI and for the large part, the
>>>>>> History UI ( i.e. the one which will display DAG details after an app has
>>>>>> disappeared ) will be dependent on YARN Timeline, the UI will not be
>>>>>> supported for Hadoop versions 2.2.x and 2.3.x given that they do not support
>>>>>> Timeline.
>>>>>> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x -
>>>>>> this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>>>>>>
>>>>>> Also, as part of TEZ-1696, I also propose changing the default version
>>>>>> to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>>>>>>
>>>>>> Thoughts/concerns?
>>>>>>
>>>>>> thanks
>>>>>> — Hitesh
>>>>>>
>>>>
>>>
>>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Jonathan Eagles <je...@gmail.com>.
Adding to the discussion. I am using some 2.5 and 2.6
On Wed, Dec 10, 2014 at 8:02 PM, Cheolsoo Park <pi...@gmail.com> wrote:
> We're using EMR 2.4 in production, and it will take a while to upgrade to
> 2.6 depending on how fast EMR can provide us 2.6 AMI. But if that's what we
> need to fully use Tez-8, we will upgrade our cluster.
>
> On Wed, Dec 10, 2014 at 1:44 PM, Siddharth Seth <ss...@apache.org> wrote:
>>
>> +1 for changing the default to 2.6.0.
>>
>> Publishing multiple jars - do you mean per hadoop version supported ?
>> That'll likely be very difficult to use - what would the dependency be ?
>> Will there be binary compatibility issues if someone builds against tez jars
>> built against Hadoop-2.6, and then deployed against 2.2 ?
>>
>> On Mon, Dec 8, 2014 at 10:02 AM, Hitesh Shah <hi...@apache.org> wrote:
>>>
>>> Thanks for the info, Douglas. We will not be dropping support for 2.4 for
>>> sure at the moment.
>>>
>>> Anyone else have any comments in this regard?
>>>
>>> @Bikas/@Sid - what do we do about publishing jars for a release? We would
>>> need to publish 2 or 3 sets?
>>>
>>> thanks
>>> — Hitesh
>>>
>>> On Dec 4, 2014, at 12:51 PM, Moore, Douglas
>>> <Do...@thinkbiganalytics.com> wrote:
>>>
>>> > 2.4.x and beyond for the customers we have
>>> >
>>> > Sent from my iPhone
>>> >
>>> >> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>>> >>
>>> >> Hello folks
>>> >>
>>> >> Could folks who are following these mailing lists do a raise of hands
>>> >> on which versions of Hadoop you are trying to run Tez on?
>>> >>
>>> >> 1) 2.2.x
>>> >> 2) 2.3.x
>>> >> 3) 2.4.x
>>> >> 4) 2.5.x
>>> >> 5) 2.6.x
>>> >>
>>> >> Given that we are building out the UI and for the large part, the
>>> >> History UI ( i.e. the one which will display DAG details after an app has
>>> >> disappeared ) will be dependent on YARN Timeline, the UI will not be
>>> >> supported for Hadoop versions 2.2.x and 2.3.x given that they do not support
>>> >> Timeline.
>>> >> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x -
>>> >> this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>>> >>
>>> >> Also, as part of TEZ-1696, I also propose changing the default version
>>> >> to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>>> >>
>>> >> Thoughts/concerns?
>>> >>
>>> >> thanks
>>> >> — Hitesh
>>> >>
>>>
>>
>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Jonathan Eagles <je...@gmail.com>.
Adding to the discussion. I am using some 2.5 and 2.6
On Wed, Dec 10, 2014 at 8:02 PM, Cheolsoo Park <pi...@gmail.com> wrote:
> We're using EMR 2.4 in production, and it will take a while to upgrade to
> 2.6 depending on how fast EMR can provide us 2.6 AMI. But if that's what we
> need to fully use Tez-8, we will upgrade our cluster.
>
> On Wed, Dec 10, 2014 at 1:44 PM, Siddharth Seth <ss...@apache.org> wrote:
>>
>> +1 for changing the default to 2.6.0.
>>
>> Publishing multiple jars - do you mean per hadoop version supported ?
>> That'll likely be very difficult to use - what would the dependency be ?
>> Will there be binary compatibility issues if someone builds against tez jars
>> built against Hadoop-2.6, and then deployed against 2.2 ?
>>
>> On Mon, Dec 8, 2014 at 10:02 AM, Hitesh Shah <hi...@apache.org> wrote:
>>>
>>> Thanks for the info, Douglas. We will not be dropping support for 2.4 for
>>> sure at the moment.
>>>
>>> Anyone else have any comments in this regard?
>>>
>>> @Bikas/@Sid - what do we do about publishing jars for a release? We would
>>> need to publish 2 or 3 sets?
>>>
>>> thanks
>>> — Hitesh
>>>
>>> On Dec 4, 2014, at 12:51 PM, Moore, Douglas
>>> <Do...@thinkbiganalytics.com> wrote:
>>>
>>> > 2.4.x and beyond for the customers we have
>>> >
>>> > Sent from my iPhone
>>> >
>>> >> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>>> >>
>>> >> Hello folks
>>> >>
>>> >> Could folks who are following these mailing lists do a raise of hands
>>> >> on which versions of Hadoop you are trying to run Tez on?
>>> >>
>>> >> 1) 2.2.x
>>> >> 2) 2.3.x
>>> >> 3) 2.4.x
>>> >> 4) 2.5.x
>>> >> 5) 2.6.x
>>> >>
>>> >> Given that we are building out the UI and for the large part, the
>>> >> History UI ( i.e. the one which will display DAG details after an app has
>>> >> disappeared ) will be dependent on YARN Timeline, the UI will not be
>>> >> supported for Hadoop versions 2.2.x and 2.3.x given that they do not support
>>> >> Timeline.
>>> >> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x -
>>> >> this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>>> >>
>>> >> Also, as part of TEZ-1696, I also propose changing the default version
>>> >> to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>>> >>
>>> >> Thoughts/concerns?
>>> >>
>>> >> thanks
>>> >> — Hitesh
>>> >>
>>>
>>
>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Cheolsoo Park <pi...@gmail.com>.
We're using EMR 2.4 in production, and it will take a while to upgrade to
2.6 depending on how fast EMR can provide us 2.6 AMI. But if that's what we
need to fully use Tez-8, we will upgrade our cluster.
On Wed, Dec 10, 2014 at 1:44 PM, Siddharth Seth <ss...@apache.org> wrote:
> +1 for changing the default to 2.6.0.
>
> Publishing multiple jars - do you mean per hadoop version supported ?
> That'll likely be very difficult to use - what would the dependency be ?
> Will there be binary compatibility issues if someone builds against tez
> jars built against Hadoop-2.6, and then deployed against 2.2 ?
>
> On Mon, Dec 8, 2014 at 10:02 AM, Hitesh Shah <hi...@apache.org> wrote:
>
>> Thanks for the info, Douglas. We will not be dropping support for 2.4 for
>> sure at the moment.
>>
>> Anyone else have any comments in this regard?
>>
>> @Bikas/@Sid - what do we do about publishing jars for a release? We would
>> need to publish 2 or 3 sets?
>>
>> thanks
>> — Hitesh
>>
>> On Dec 4, 2014, at 12:51 PM, Moore, Douglas <
>> Douglas.Moore@thinkbiganalytics.com> wrote:
>>
>> > 2.4.x and beyond for the customers we have
>> >
>> > Sent from my iPhone
>> >
>> >> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>> >>
>> >> Hello folks
>> >>
>> >> Could folks who are following these mailing lists do a raise of hands
>> on which versions of Hadoop you are trying to run Tez on?
>> >>
>> >> 1) 2.2.x
>> >> 2) 2.3.x
>> >> 3) 2.4.x
>> >> 4) 2.5.x
>> >> 5) 2.6.x
>> >>
>> >> Given that we are building out the UI and for the large part, the
>> History UI ( i.e. the one which will display DAG details after an app has
>> disappeared ) will be dependent on YARN Timeline, the UI will not be
>> supported for Hadoop versions 2.2.x and 2.3.x given that they do not
>> support Timeline.
>> >> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x -
>> this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>> >>
>> >> Also, as part of TEZ-1696, I also propose changing the default version
>> to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>> >>
>> >> Thoughts/concerns?
>> >>
>> >> thanks
>> >> — Hitesh
>> >>
>>
>>
>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Cheolsoo Park <pi...@gmail.com>.
We're using EMR 2.4 in production, and it will take a while to upgrade to
2.6 depending on how fast EMR can provide us 2.6 AMI. But if that's what we
need to fully use Tez-8, we will upgrade our cluster.
On Wed, Dec 10, 2014 at 1:44 PM, Siddharth Seth <ss...@apache.org> wrote:
> +1 for changing the default to 2.6.0.
>
> Publishing multiple jars - do you mean per hadoop version supported ?
> That'll likely be very difficult to use - what would the dependency be ?
> Will there be binary compatibility issues if someone builds against tez
> jars built against Hadoop-2.6, and then deployed against 2.2 ?
>
> On Mon, Dec 8, 2014 at 10:02 AM, Hitesh Shah <hi...@apache.org> wrote:
>
>> Thanks for the info, Douglas. We will not be dropping support for 2.4 for
>> sure at the moment.
>>
>> Anyone else have any comments in this regard?
>>
>> @Bikas/@Sid - what do we do about publishing jars for a release? We would
>> need to publish 2 or 3 sets?
>>
>> thanks
>> — Hitesh
>>
>> On Dec 4, 2014, at 12:51 PM, Moore, Douglas <
>> Douglas.Moore@thinkbiganalytics.com> wrote:
>>
>> > 2.4.x and beyond for the customers we have
>> >
>> > Sent from my iPhone
>> >
>> >> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>> >>
>> >> Hello folks
>> >>
>> >> Could folks who are following these mailing lists do a raise of hands
>> on which versions of Hadoop you are trying to run Tez on?
>> >>
>> >> 1) 2.2.x
>> >> 2) 2.3.x
>> >> 3) 2.4.x
>> >> 4) 2.5.x
>> >> 5) 2.6.x
>> >>
>> >> Given that we are building out the UI and for the large part, the
>> History UI ( i.e. the one which will display DAG details after an app has
>> disappeared ) will be dependent on YARN Timeline, the UI will not be
>> supported for Hadoop versions 2.2.x and 2.3.x given that they do not
>> support Timeline.
>> >> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x -
>> this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>> >>
>> >> Also, as part of TEZ-1696, I also propose changing the default version
>> to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>> >>
>> >> Thoughts/concerns?
>> >>
>> >> thanks
>> >> — Hitesh
>> >>
>>
>>
>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Siddharth Seth <ss...@apache.org>.
+1 for changing the default to 2.6.0.
Publishing multiple jars - do you mean per hadoop version supported ?
That'll likely be very difficult to use - what would the dependency be ?
Will there be binary compatibility issues if someone builds against tez
jars built against Hadoop-2.6, and then deployed against 2.2 ?
On Mon, Dec 8, 2014 at 10:02 AM, Hitesh Shah <hi...@apache.org> wrote:
> Thanks for the info, Douglas. We will not be dropping support for 2.4 for
> sure at the moment.
>
> Anyone else have any comments in this regard?
>
> @Bikas/@Sid - what do we do about publishing jars for a release? We would
> need to publish 2 or 3 sets?
>
> thanks
> — Hitesh
>
> On Dec 4, 2014, at 12:51 PM, Moore, Douglas <
> Douglas.Moore@thinkbiganalytics.com> wrote:
>
> > 2.4.x and beyond for the customers we have
> >
> > Sent from my iPhone
> >
> >> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
> >>
> >> Hello folks
> >>
> >> Could folks who are following these mailing lists do a raise of hands
> on which versions of Hadoop you are trying to run Tez on?
> >>
> >> 1) 2.2.x
> >> 2) 2.3.x
> >> 3) 2.4.x
> >> 4) 2.5.x
> >> 5) 2.6.x
> >>
> >> Given that we are building out the UI and for the large part, the
> History UI ( i.e. the one which will display DAG details after an app has
> disappeared ) will be dependent on YARN Timeline, the UI will not be
> supported for Hadoop versions 2.2.x and 2.3.x given that they do not
> support Timeline.
> >> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x -
> this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
> >>
> >> Also, as part of TEZ-1696, I also propose changing the default version
> to 2.6.0 given that secure Timeline is only available as of 2.6.0.
> >>
> >> Thoughts/concerns?
> >>
> >> thanks
> >> — Hitesh
> >>
>
>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Hitesh Shah <hi...@apache.org>.
Thanks for the info Johannes.
Given the absence of timeline and therefore no historical data, it is unlikely that a tez history UI will be supported against 2.2.
That said, there is probably going to be future work to build a UI into the application master itself for tracking a running DAG which likely should not be timeline dependent.
— HItesh
On Dec 9, 2014, at 8:29 PM, Johannes Zillmann <jz...@googlemail.com> wrote:
> We have a wide range of customers… so basically all of them ;)
> Anyhow would be good to have at least clear announcements like for 0.5 there is 2.2 minimum, for 0.6 there is… etc..
> If the Hadoop UI integration could be delivered with 2.2 that would be awesome!
>
> best
> Johannes
>
>> On 08 Dec 2014, at 19:02, Hitesh Shah <hi...@apache.org> wrote:
>>
>> Thanks for the info, Douglas. We will not be dropping support for 2.4 for sure at the moment.
>>
>> Anyone else have any comments in this regard?
>>
>> @Bikas/@Sid - what do we do about publishing jars for a release? We would need to publish 2 or 3 sets?
>>
>> thanks
>> — Hitesh
>>
>> On Dec 4, 2014, at 12:51 PM, Moore, Douglas <Do...@thinkbiganalytics.com> wrote:
>>
>>> 2.4.x and beyond for the customers we have
>>>
>>> Sent from my iPhone
>>>
>>>> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>>>>
>>>> Hello folks
>>>>
>>>> Could folks who are following these mailing lists do a raise of hands on which versions of Hadoop you are trying to run Tez on?
>>>>
>>>> 1) 2.2.x
>>>> 2) 2.3.x
>>>> 3) 2.4.x
>>>> 4) 2.5.x
>>>> 5) 2.6.x
>>>>
>>>> Given that we are building out the UI and for the large part, the History UI ( i.e. the one which will display DAG details after an app has disappeared ) will be dependent on YARN Timeline, the UI will not be supported for Hadoop versions 2.2.x and 2.3.x given that they do not support Timeline.
>>>> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x - this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>>>>
>>>> Also, as part of TEZ-1696, I also propose changing the default version to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>>>>
>>>> Thoughts/concerns?
>>>>
>>>> thanks
>>>> — Hitesh
>>>>
>>
>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Hitesh Shah <hi...@apache.org>.
Thanks for the info Johannes.
Given the absence of timeline and therefore no historical data, it is unlikely that a tez history UI will be supported against 2.2.
That said, there is probably going to be future work to build a UI into the application master itself for tracking a running DAG which likely should not be timeline dependent.
— HItesh
On Dec 9, 2014, at 8:29 PM, Johannes Zillmann <jz...@googlemail.com> wrote:
> We have a wide range of customers… so basically all of them ;)
> Anyhow would be good to have at least clear announcements like for 0.5 there is 2.2 minimum, for 0.6 there is… etc..
> If the Hadoop UI integration could be delivered with 2.2 that would be awesome!
>
> best
> Johannes
>
>> On 08 Dec 2014, at 19:02, Hitesh Shah <hi...@apache.org> wrote:
>>
>> Thanks for the info, Douglas. We will not be dropping support for 2.4 for sure at the moment.
>>
>> Anyone else have any comments in this regard?
>>
>> @Bikas/@Sid - what do we do about publishing jars for a release? We would need to publish 2 or 3 sets?
>>
>> thanks
>> — Hitesh
>>
>> On Dec 4, 2014, at 12:51 PM, Moore, Douglas <Do...@thinkbiganalytics.com> wrote:
>>
>>> 2.4.x and beyond for the customers we have
>>>
>>> Sent from my iPhone
>>>
>>>> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>>>>
>>>> Hello folks
>>>>
>>>> Could folks who are following these mailing lists do a raise of hands on which versions of Hadoop you are trying to run Tez on?
>>>>
>>>> 1) 2.2.x
>>>> 2) 2.3.x
>>>> 3) 2.4.x
>>>> 4) 2.5.x
>>>> 5) 2.6.x
>>>>
>>>> Given that we are building out the UI and for the large part, the History UI ( i.e. the one which will display DAG details after an app has disappeared ) will be dependent on YARN Timeline, the UI will not be supported for Hadoop versions 2.2.x and 2.3.x given that they do not support Timeline.
>>>> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x - this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>>>>
>>>> Also, as part of TEZ-1696, I also propose changing the default version to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>>>>
>>>> Thoughts/concerns?
>>>>
>>>> thanks
>>>> — Hitesh
>>>>
>>
>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by "Moore, Douglas" <Do...@thinkbiganalytics.com>.
To make it easier, nearly all of our customers depend upon commercial
support.
Unless you get the major vendors to build patches for this feature,
supporting the older versions may be a mute point.
On 12/9/14 11:29 PM, "Johannes Zillmann" <jz...@googlemail.com> wrote:
>We have a wide range of customersŠ so basically all of them ;)
>Anyhow would be good to have at least clear announcements like for 0.5
>there is 2.2 minimum, for 0.6 there isŠ etc..
>If the Hadoop UI integration could be delivered with 2.2 that would be
>awesome!
>
>best
>Johannes
>
>> On 08 Dec 2014, at 19:02, Hitesh Shah <hi...@apache.org> wrote:
>>
>> Thanks for the info, Douglas. We will not be dropping support for 2.4
>>for sure at the moment.
>>
>> Anyone else have any comments in this regard?
>>
>> @Bikas/@Sid - what do we do about publishing jars for a release? We
>>would need to publish 2 or 3 sets?
>>
>> thanks
>> ‹ Hitesh
>>
>> On Dec 4, 2014, at 12:51 PM, Moore, Douglas
>><Do...@thinkbiganalytics.com> wrote:
>>
>>> 2.4.x and beyond for the customers we have
>>>
>>> Sent from my iPhone
>>>
>>>> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>>>>
>>>> Hello folks
>>>>
>>>> Could folks who are following these mailing lists do a raise of hands
>>>>on which versions of Hadoop you are trying to run Tez on?
>>>>
>>>> 1) 2.2.x
>>>> 2) 2.3.x
>>>> 3) 2.4.x
>>>> 4) 2.5.x
>>>> 5) 2.6.x
>>>>
>>>> Given that we are building out the UI and for the large part, the
>>>>History UI ( i.e. the one which will display DAG details after an app
>>>>has disappeared ) will be dependent on YARN Timeline, the UI will not
>>>>be supported for Hadoop versions 2.2.x and 2.3.x given that they do
>>>>not support Timeline.
>>>> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x -
>>>>this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>>>>
>>>> Also, as part of TEZ-1696, I also propose changing the default
>>>>version to 2.6.0 given that secure Timeline is only available as of
>>>>2.6.0.
>>>>
>>>> Thoughts/concerns?
>>>>
>>>> thanks
>>>> ‹ Hitesh
>>>>
>>
>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by "Moore, Douglas" <Do...@thinkbiganalytics.com>.
To make it easier, nearly all of our customers depend upon commercial
support.
Unless you get the major vendors to build patches for this feature,
supporting the older versions may be a mute point.
On 12/9/14 11:29 PM, "Johannes Zillmann" <jz...@googlemail.com> wrote:
>We have a wide range of customersŠ so basically all of them ;)
>Anyhow would be good to have at least clear announcements like for 0.5
>there is 2.2 minimum, for 0.6 there isŠ etc..
>If the Hadoop UI integration could be delivered with 2.2 that would be
>awesome!
>
>best
>Johannes
>
>> On 08 Dec 2014, at 19:02, Hitesh Shah <hi...@apache.org> wrote:
>>
>> Thanks for the info, Douglas. We will not be dropping support for 2.4
>>for sure at the moment.
>>
>> Anyone else have any comments in this regard?
>>
>> @Bikas/@Sid - what do we do about publishing jars for a release? We
>>would need to publish 2 or 3 sets?
>>
>> thanks
>> ‹ Hitesh
>>
>> On Dec 4, 2014, at 12:51 PM, Moore, Douglas
>><Do...@thinkbiganalytics.com> wrote:
>>
>>> 2.4.x and beyond for the customers we have
>>>
>>> Sent from my iPhone
>>>
>>>> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>>>>
>>>> Hello folks
>>>>
>>>> Could folks who are following these mailing lists do a raise of hands
>>>>on which versions of Hadoop you are trying to run Tez on?
>>>>
>>>> 1) 2.2.x
>>>> 2) 2.3.x
>>>> 3) 2.4.x
>>>> 4) 2.5.x
>>>> 5) 2.6.x
>>>>
>>>> Given that we are building out the UI and for the large part, the
>>>>History UI ( i.e. the one which will display DAG details after an app
>>>>has disappeared ) will be dependent on YARN Timeline, the UI will not
>>>>be supported for Hadoop versions 2.2.x and 2.3.x given that they do
>>>>not support Timeline.
>>>> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x -
>>>>this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>>>>
>>>> Also, as part of TEZ-1696, I also propose changing the default
>>>>version to 2.6.0 given that secure Timeline is only available as of
>>>>2.6.0.
>>>>
>>>> Thoughts/concerns?
>>>>
>>>> thanks
>>>> ‹ Hitesh
>>>>
>>
>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Johannes Zillmann <jz...@googlemail.com>.
We have a wide range of customers… so basically all of them ;)
Anyhow would be good to have at least clear announcements like for 0.5 there is 2.2 minimum, for 0.6 there is… etc..
If the Hadoop UI integration could be delivered with 2.2 that would be awesome!
best
Johannes
> On 08 Dec 2014, at 19:02, Hitesh Shah <hi...@apache.org> wrote:
>
> Thanks for the info, Douglas. We will not be dropping support for 2.4 for sure at the moment.
>
> Anyone else have any comments in this regard?
>
> @Bikas/@Sid - what do we do about publishing jars for a release? We would need to publish 2 or 3 sets?
>
> thanks
> — Hitesh
>
> On Dec 4, 2014, at 12:51 PM, Moore, Douglas <Do...@thinkbiganalytics.com> wrote:
>
>> 2.4.x and beyond for the customers we have
>>
>> Sent from my iPhone
>>
>>> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>>>
>>> Hello folks
>>>
>>> Could folks who are following these mailing lists do a raise of hands on which versions of Hadoop you are trying to run Tez on?
>>>
>>> 1) 2.2.x
>>> 2) 2.3.x
>>> 3) 2.4.x
>>> 4) 2.5.x
>>> 5) 2.6.x
>>>
>>> Given that we are building out the UI and for the large part, the History UI ( i.e. the one which will display DAG details after an app has disappeared ) will be dependent on YARN Timeline, the UI will not be supported for Hadoop versions 2.2.x and 2.3.x given that they do not support Timeline.
>>> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x - this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>>>
>>> Also, as part of TEZ-1696, I also propose changing the default version to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>>>
>>> Thoughts/concerns?
>>>
>>> thanks
>>> — Hitesh
>>>
>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Siddharth Seth <ss...@apache.org>.
+1 for changing the default to 2.6.0.
Publishing multiple jars - do you mean per hadoop version supported ?
That'll likely be very difficult to use - what would the dependency be ?
Will there be binary compatibility issues if someone builds against tez
jars built against Hadoop-2.6, and then deployed against 2.2 ?
On Mon, Dec 8, 2014 at 10:02 AM, Hitesh Shah <hi...@apache.org> wrote:
> Thanks for the info, Douglas. We will not be dropping support for 2.4 for
> sure at the moment.
>
> Anyone else have any comments in this regard?
>
> @Bikas/@Sid - what do we do about publishing jars for a release? We would
> need to publish 2 or 3 sets?
>
> thanks
> — Hitesh
>
> On Dec 4, 2014, at 12:51 PM, Moore, Douglas <
> Douglas.Moore@thinkbiganalytics.com> wrote:
>
> > 2.4.x and beyond for the customers we have
> >
> > Sent from my iPhone
> >
> >> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
> >>
> >> Hello folks
> >>
> >> Could folks who are following these mailing lists do a raise of hands
> on which versions of Hadoop you are trying to run Tez on?
> >>
> >> 1) 2.2.x
> >> 2) 2.3.x
> >> 3) 2.4.x
> >> 4) 2.5.x
> >> 5) 2.6.x
> >>
> >> Given that we are building out the UI and for the large part, the
> History UI ( i.e. the one which will display DAG details after an app has
> disappeared ) will be dependent on YARN Timeline, the UI will not be
> supported for Hadoop versions 2.2.x and 2.3.x given that they do not
> support Timeline.
> >> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x -
> this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
> >>
> >> Also, as part of TEZ-1696, I also propose changing the default version
> to 2.6.0 given that secure Timeline is only available as of 2.6.0.
> >>
> >> Thoughts/concerns?
> >>
> >> thanks
> >> — Hitesh
> >>
>
>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Johannes Zillmann <jz...@googlemail.com>.
We have a wide range of customers… so basically all of them ;)
Anyhow would be good to have at least clear announcements like for 0.5 there is 2.2 minimum, for 0.6 there is… etc..
If the Hadoop UI integration could be delivered with 2.2 that would be awesome!
best
Johannes
> On 08 Dec 2014, at 19:02, Hitesh Shah <hi...@apache.org> wrote:
>
> Thanks for the info, Douglas. We will not be dropping support for 2.4 for sure at the moment.
>
> Anyone else have any comments in this regard?
>
> @Bikas/@Sid - what do we do about publishing jars for a release? We would need to publish 2 or 3 sets?
>
> thanks
> — Hitesh
>
> On Dec 4, 2014, at 12:51 PM, Moore, Douglas <Do...@thinkbiganalytics.com> wrote:
>
>> 2.4.x and beyond for the customers we have
>>
>> Sent from my iPhone
>>
>>> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>>>
>>> Hello folks
>>>
>>> Could folks who are following these mailing lists do a raise of hands on which versions of Hadoop you are trying to run Tez on?
>>>
>>> 1) 2.2.x
>>> 2) 2.3.x
>>> 3) 2.4.x
>>> 4) 2.5.x
>>> 5) 2.6.x
>>>
>>> Given that we are building out the UI and for the large part, the History UI ( i.e. the one which will display DAG details after an app has disappeared ) will be dependent on YARN Timeline, the UI will not be supported for Hadoop versions 2.2.x and 2.3.x given that they do not support Timeline.
>>> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x - this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>>>
>>> Also, as part of TEZ-1696, I also propose changing the default version to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>>>
>>> Thoughts/concerns?
>>>
>>> thanks
>>> — Hitesh
>>>
>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Hitesh Shah <hi...@apache.org>.
Thanks for the info, Douglas. We will not be dropping support for 2.4 for sure at the moment.
Anyone else have any comments in this regard?
@Bikas/@Sid - what do we do about publishing jars for a release? We would need to publish 2 or 3 sets?
thanks
— Hitesh
On Dec 4, 2014, at 12:51 PM, Moore, Douglas <Do...@thinkbiganalytics.com> wrote:
> 2.4.x and beyond for the customers we have
>
> Sent from my iPhone
>
>> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>>
>> Hello folks
>>
>> Could folks who are following these mailing lists do a raise of hands on which versions of Hadoop you are trying to run Tez on?
>>
>> 1) 2.2.x
>> 2) 2.3.x
>> 3) 2.4.x
>> 4) 2.5.x
>> 5) 2.6.x
>>
>> Given that we are building out the UI and for the large part, the History UI ( i.e. the one which will display DAG details after an app has disappeared ) will be dependent on YARN Timeline, the UI will not be supported for Hadoop versions 2.2.x and 2.3.x given that they do not support Timeline.
>> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x - this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>>
>> Also, as part of TEZ-1696, I also propose changing the default version to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>>
>> Thoughts/concerns?
>>
>> thanks
>> — Hitesh
>>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by Hitesh Shah <hi...@apache.org>.
Thanks for the info, Douglas. We will not be dropping support for 2.4 for sure at the moment.
Anyone else have any comments in this regard?
@Bikas/@Sid - what do we do about publishing jars for a release? We would need to publish 2 or 3 sets?
thanks
— Hitesh
On Dec 4, 2014, at 12:51 PM, Moore, Douglas <Do...@thinkbiganalytics.com> wrote:
> 2.4.x and beyond for the customers we have
>
> Sent from my iPhone
>
>> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>>
>> Hello folks
>>
>> Could folks who are following these mailing lists do a raise of hands on which versions of Hadoop you are trying to run Tez on?
>>
>> 1) 2.2.x
>> 2) 2.3.x
>> 3) 2.4.x
>> 4) 2.5.x
>> 5) 2.6.x
>>
>> Given that we are building out the UI and for the large part, the History UI ( i.e. the one which will display DAG details after an app has disappeared ) will be dependent on YARN Timeline, the UI will not be supported for Hadoop versions 2.2.x and 2.3.x given that they do not support Timeline.
>> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x - this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>>
>> Also, as part of TEZ-1696, I also propose changing the default version to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>>
>> Thoughts/concerns?
>>
>> thanks
>> — Hitesh
>>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by "Moore, Douglas" <Do...@thinkbiganalytics.com>.
2.4.x and beyond for the customers we have
Sent from my iPhone
> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>
> Hello folks
>
> Could folks who are following these mailing lists do a raise of hands on which versions of Hadoop you are trying to run Tez on?
>
> 1) 2.2.x
> 2) 2.3.x
> 3) 2.4.x
> 4) 2.5.x
> 5) 2.6.x
>
> Given that we are building out the UI and for the large part, the History UI ( i.e. the one which will display DAG details after an app has disappeared ) will be dependent on YARN Timeline, the UI will not be supported for Hadoop versions 2.2.x and 2.3.x given that they do not support Timeline.
> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x - this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>
> Also, as part of TEZ-1696, I also propose changing the default version to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>
> Thoughts/concerns?
>
> thanks
> — Hitesh
>
Re: [DISCUSS] What versions of hadoop should we support?
Posted by "Moore, Douglas" <Do...@thinkbiganalytics.com>.
2.4.x and beyond for the customers we have
Sent from my iPhone
> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hi...@apache.org> wrote:
>
> Hello folks
>
> Could folks who are following these mailing lists do a raise of hands on which versions of Hadoop you are trying to run Tez on?
>
> 1) 2.2.x
> 2) 2.3.x
> 3) 2.4.x
> 4) 2.5.x
> 5) 2.6.x
>
> Given that we are building out the UI and for the large part, the History UI ( i.e. the one which will display DAG details after an app has disappeared ) will be dependent on YARN Timeline, the UI will not be supported for Hadoop versions 2.2.x and 2.3.x given that they do not support Timeline.
> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x - this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>
> Also, as part of TEZ-1696, I also propose changing the default version to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>
> Thoughts/concerns?
>
> thanks
> — Hitesh
>