[jira] Commented: (PIG-18) Making Pig work with HOD 0.4

["Craig Macdonald (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572106#action_12572106 ] 

Craig Macdonald commented on PIG-18:
------------------------------------

I had a think on this issue. It's fairly easy to integrate HOD, if the ssh gateway support is *not* required. This is because, essentially, we do create a local temporary directory, then call HOD. 

For the ssh gateway support, there needs to be a way to get the Hadoop configuration across the connection. How important a use-case is this for Yahoo? 
Feasible solutions:
1. the current expect script parsing HOD output.
2. Using a shell script wrapper emulating HOD? This would call hod on the remote machine, and then scp the resulting configuration back
3. Another option would be to add an option to HOD that allows the Hadoop XML config file to be displayed once nodes have been allocated.
4. You install qsub on all machines that run Pig.
5. You write a shell script to emulate qsub using ssh and use HOD locally.
{noformat}
#!/bin/bash
GATEWAY=headnode
exec ssh headnode qsub "$@"
{noformat}

Craig

> Making Pig work with HOD 0.4
> ----------------------------
>
>                 Key: PIG-18
>                 URL: https://issues.apache.org/jira/browse/PIG-18
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Olga Natkovich
>            Assignee: Olga Natkovich
>            Priority: Critical
>
> Pig needs to work with the soon to be released hadoop on demand (HOD) version.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.