Re: Cascading 3.0 WIP with Apache Tez platform support

[Chris K Wensel <ch...@wensel.net>]

Hey all

Just a heads up we are up to wip-72 now.

This moves Tez to 0.6.0, and changes the default Hadoop version to 2.6. Tez intends to remain compatible with Hadoop 2.4. So from a runtime perspective, you shouldn’t see a difference.

This also provides deep counter support to the task/slice level. This in turn requires a YARN history server to be running, things will gracefully downgrade without it.

I’ve also updated the README to include help on using Tez on Amazon EMR

https://github.com/cwensel/cascading/blob/wip-3.0/cascading-hadoop2-tez/README.md <https://github.com/cwensel/cascading/blob/wip-3.0/cascading-hadoop2-tez/README.md>

As a convenience we are hosting Tez binaries in our S3 buckets. 

https://github.com/cwensel/cascading/blob/wip-3.0/cascading-hadoop2-tez/README.md#tez-binaries <https://github.com/cwensel/cascading/blob/wip-3.0/cascading-hadoop2-tez/README.md#tez-binaries>

Lastly, we only have a few more items to build out for Tez listed below before we are feature complete for 3.0

http://docs.cascading.org/roadmap/cascading/cascading-3.html <http://docs.cascading.org/roadmap/cascading/cascading-3.html>

If you haven’t done so, please test your apps and languages on 3.0 so we can address any issues, especially API changes, before 3.0 final.

We are also publishing 3.0 Fluid artifacts as well.

http://www.cascading.org/fluid/ <http://www.cascading.org/fluid/>


ckw


> On Sep 17, 2014, at 11:28 AM, Chris K Wensel <ch...@wensel.net> wrote:
> 
> 
> Hey all
> 
> Happy to let everyone know we just pushed out a new 3.0 WIP that supports Apache Tez as a runtime platform.
> 
> The official announcement is here:
> 
>   http://www.cascading.org/2014/09/17/cascading-3-0-wip-now-supports-apache-tez/ <http://www.cascading.org/2014/09/17/cascading-3-0-wip-now-supports-apache-tez/>
> 
> Note the wip-50 download doesn't include hadoop2-tez.jar, it will show up in wip-51. But conjars has the binaries.
> 
>   http://conjars.org/cascading/cascading-hadoop2-tez <http://conjars.org/cascading/cascading-hadoop2-tez>
> 
> And the JavaDoc can be found here:
>   
>   http://docs.concurrentinc.com/cascading/3.0/javadoc/cascading-hadoop2-tez/ <http://docs.concurrentinc.com/cascading/3.0/javadoc/cascading-hadoop2-tez/>
> 
> The current working wip branch:
> 
>   https://github.com/cwensel/cascading/tree/wip-3.0 <https://github.com/cwensel/cascading/tree/wip-3.0>
> 
> Notes and knowns issues can be tracked here:
> 
>   https://github.com/cwensel/cascading/tree/wip-3.0/cascading-hadoop2-tez <https://github.com/cwensel/cascading/tree/wip-3.0/cascading-hadoop2-tez>
> 
> Know all platform tests do pass, and the resulting binaries have been pushed to conjars.org <http://conjars.org/>.
> 
> That said, we expect there to be issues, so please email the cascading-user list with any questions or bugs.
> 
>   http://www.cascading.org/support/ <http://www.cascading.org/support/>
> 
> We have also updated the sample apps to use Tez, other projects will also be updated as well over time.
> 
>   https://github.com/Cascading/cascading.samples/tree/wip-3.0 <https://github.com/Cascading/cascading.samples/tree/wip-3.0>
> 
> cheers,
> chris
> 
> --
> Chris K Wensel
> chris@concurrentinc.com <ma...@concurrentinc.com>
> http://concurrentinc.com <http://concurrentinc.com/>

—
Chris K Wensel
chris@wensel.net