Need example of python code with dependency files

[Ryan Compton <co...@gmail.com>]

I have some python code I'd like to deploy with a pig script. The .py
code takes input from sys.stdin and outputs to sys.stdout. It also
needs some parameter files to run properly.

The book "Programming Pig" tells me:

"The workaround for this is to create a TAR file and ship that, and
then have a step in your executable that unbundles the TAR file"

but I'm still real confused. I need an example of how to unpack it (eg
unpack in a bash script I pass around, unpack in the pig script, ??)

Re: Need example of python code with dependency files

[Ryan Compton <co...@gmail.com>]

Thanks, this worked for some test programs, but then I ran into some
other problems when I had to import things (for some reason pig was
looking for python2.4 stuff even though I explicitly told it 2.7, I'm
assuming this is more of a CentOS/Jython issue).

I ended up using PyInstaller to package up my code and streammed though that.

On Wed, Nov 6, 2013 at 6:23 AM,  <wi...@thomsonreuters.com> wrote:
> You said "The .py code takes input from sys.stdin and outputs to sys.stdout" so I infer you are talking about streaming, not a python UDF. In that case, rather than streaming through your python script P.py, instead stream through a shell script S.sh. The shell script can untar shipped or cached files, then run P.py.  Maybe not completely elegant, but this approach works for me.
>
>
> William F Dowling
> Senior Technologist
> Thomson Reuters
>
> -----Original Message-----
> From: Ryan Compton [mailto:compton.ryan@gmail.com]
> Sent: Tuesday, November 05, 2013 6:40 PM
> To: user@pig.apache.org
> Subject: Need example of python code with dependency files
>
> I have some python code I'd like to deploy with a pig script. The .py
> code takes input from sys.stdin and outputs to sys.stdout. It also
> needs some parameter files to run properly.
>
> The book "Programming Pig" tells me:
>
> "The workaround for this is to create a TAR file and ship that, and
> then have a step in your executable that unbundles the TAR file"
>
> but I'm still real confused. I need an example of how to unpack it (eg
> unpack in a bash script I pass around, unpack in the pig script, ??)

RE: Need example of python code with dependency files

[wi...@thomsonreuters.com]

You said "The .py code takes input from sys.stdin and outputs to sys.stdout" so I infer you are talking about streaming, not a python UDF. In that case, rather than streaming through your python script P.py, instead stream through a shell script S.sh. The shell script can untar shipped or cached files, then run P.py.  Maybe not completely elegant, but this approach works for me.


William F Dowling
Senior Technologist
Thomson Reuters

-----Original Message-----
From: Ryan Compton [mailto:compton.ryan@gmail.com] 
Sent: Tuesday, November 05, 2013 6:40 PM
To: user@pig.apache.org
Subject: Need example of python code with dependency files

I have some python code I'd like to deploy with a pig script. The .py
code takes input from sys.stdin and outputs to sys.stdout. It also
needs some parameter files to run properly.

The book "Programming Pig" tells me:

"The workaround for this is to create a TAR file and ship that, and
then have a step in your executable that unbundles the TAR file"

but I'm still real confused. I need an example of how to unpack it (eg
unpack in a bash script I pass around, unpack in the pig script, ??)