Pattern for pig/hive setup files

[Pradeep Gollakota <pr...@gmail.com>]

Hi All,

I'm trying to establish a good pattern and practice with Oozie for sharing
a setup file for pig/hive. For example, I have several scripts that use a
set of UDFs that are built in-house. In order to use the UDF, I need to add
the jar file, and then register the UDF. Rather than repeating this process
multiple times, I can simply put this in the .hiverc/.pigbootup file of the
user which will be running these scripts. However, this does not translate
well into Oozieland. Another option is to put the common code into a
"setup" file and add "source setup" to the beginning of every script. This
is kind of ugly in that each file will have the same "source setup"
command. There's another problem of the path'ing not necessarily being
translatable across scrips/workflows.

Has anyone come up with a good pattern for sharing hive/pig code across
scripts in Oozieland?

Thanks,
Pradeep

Re: Pattern for pig/hive setup files

[Edward Capriolo <ed...@gmail.com>]

Newer versions of hive allow permanent and groovy defined functions.  I
know that does not help your pig case.

On Tue, Jan 27, 2015 at 5:08 PM, Pradeep Gollakota <pr...@gmail.com>
wrote:

> Hi All,
>
> I'm trying to establish a good pattern and practice with Oozie for sharing
> a setup file for pig/hive. For example, I have several scripts that use a
> set of UDFs that are built in-house. In order to use the UDF, I need to add
> the jar file, and then register the UDF. Rather than repeating this process
> multiple times, I can simply put this in the .hiverc/.pigbootup file of the
> user which will be running these scripts. However, this does not translate
> well into Oozieland. Another option is to put the common code into a
> "setup" file and add "source setup" to the beginning of every script. This
> is kind of ugly in that each file will have the same "source setup"
> command. There's another problem of the path'ing not necessarily being
> translatable across scrips/workflows.
>
> Has anyone come up with a good pattern for sharing hive/pig code across
> scripts in Oozieland?
>
> Thanks,
> Pradeep
>