precompiler directives?

[Kevin Burton <bu...@spinn3r.com>]

I seem to have a need for pre compiler directives in pig.

These aren't part of the compiled map reduce job….

For example:

if file_exists( "/foo" ):
  run prepare.pig

run execute.pig

…. the prepare.pig creates files that execute.pig needs… I *could* move this
outside of the script but then pig can't see this and can't potentially
optimize things.

-- 

Founder/CEO Spinn3r.com

Location: *San Francisco, CA*
Skype: *burtonator*

Skype-in: *(415) 871-0687*

Re: precompiler directives?

[Alan Gates <ga...@hortonworks.com>]

Supporting use cases like this is why we added Python embedding in 0.9.  You can still make use of the optimizer by building the script based on the outcome of the file's existence (forgive the probably incorrect Python syntax as I'm not a heavy Python user, but hopefully the ideas are clear):

if file_exists("/foo"):
	script = "..."

script += "..." # add in part you run no matter what
Pig.compile(script).bind().run()

Alan.

On Aug 24, 2011, at 12:27 AM, Kevin Burton wrote:

> I seem to have a need for pre compiler directives in pig.
> 
> These aren't part of the compiled map reduce job….
> 
> For example:
> 
> if file_exists( "/foo" ):
>  run prepare.pig
> 
> run execute.pig
> 
> …. the prepare.pig creates files that execute.pig needs… I *could* move this
> outside of the script but then pig can't see this and can't potentially
> optimize things.
> 
> -- 
> 
> Founder/CEO Spinn3r.com
> 
> Location: *San Francisco, CA*
> Skype: *burtonator*
> 
> Skype-in: *(415) 871-0687*