You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pig.apache.org by Doug Daniels <dd...@mortardata.com> on 2011/11/20 18:28:10 UTC

Streaming UDFs

We've been using pig's jython UDF support and really enjoying it, but we're finding several cases where we need python modules with C extensions, which jython doesn't support.

While we could use the STREAM operator to make this work, it'd be great to have the simplicity, type-checking/casting, and exact-field-using of UDFs.   I think we could get that by adding Streaming UDFs, for which I've sketched an idea on the wiki: https://cwiki.apache.org/confluence/display/PIG/StreamingUDFs

It's still just a sketch, but I'd love feedback on the direction, or any other ideas if people have thought about it in the past.

Best,
Doug