You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Harish Gopalan <ha...@gmail.com> on 2016/05/10 14:27:37 UTC
Retrieving Pig script from MR job config
Hi,
Is it possible to retrieve the original pig script from the MR job
configuration similar to hive i.e hive.query.string.
I tried to deserialize pig.pigContext value from the MR job configuration
but couldn't get it ?
Please advise.
Regards
Harish
Re: Retrieving Pig script from MR job config
Posted by Rohini Palaniswamy <ro...@gmail.com>.
It should be there in the job conf (
https://github.com/apache/pig/blob/branch-0.12/src/org/apache/pig/tools/pigstats/ScriptState.java#L112).
I have used it even with Pig 0.10.
You can use pig.logical.plan.signature configuration. Refer
https://issues.apache.org/jira/browse/PIG-2587.
On Sat, May 28, 2016 at 5:21 AM, Harish Gopalan <ha...@gmail.com>
wrote:
> Hi Rohini,
>
> Thanks. But I couldn't find any property such as pig.script in my current
> Pig version i.e pig 0.12 , the closest I could find is pig.script.id , if
> not the pig script , is there a way for me to fingerprint every pig script
> that is run on the system i.e YARN ? In other words I would like to find
> out recurrent pig job executions provided it is the same code that is
> executing. I guess I have to match it by retrieving the Abstract Syntax
> tree but not very sure.
>
> Regards
>
> Harish
> On 5/27/2016 11:09 PM, Rohini Palaniswamy wrote:
>
> You can find the pig script in pig.script setting. It is base64 encoded and
> you will have to decode it. If the script is too long, it will be truncated
> to 10K lines.
>
> Regards,
> Rohini
>
> On Tue, May 10, 2016 at 7:27 AM, Harish Gopalan
> <ha...@gmail.com> <ha...@gmail.com>
> wrote:
>
>
> Hi,
>
> Is it possible to retrieve the original pig script from the MR job
> configuration similar to hive i.e hive.query.string.
>
> I tried to deserialize pig.pigContext value from the MR job configuration
> but couldn't get it ?
>
> Please advise.
>
> Regards
> Harish
>
Re: Retrieving Pig script from MR job config
Posted by Harish Gopalan <ha...@gmail.com>.
Hi Rohini,
Thanks. But I couldn't find any property such as pig.script in my current
Pig version i.e pig 0.12 , the closest I could find is pig.script.id , if
not the pig script , is there a way for me to fingerprint every pig script
that is run on the system i.e YARN ? In other words I would like to find
out recurrent pig job executions provided it is the same code that is
executing. I guess I have to match it by retrieving the Abstract Syntax
tree but not very sure.
Regards
Harish
On 5/27/2016 11:09 PM, Rohini Palaniswamy wrote:
You can find the pig script in pig.script setting. It is base64 encoded and
you will have to decode it. If the script is too long, it will be truncated
to 10K lines.
Regards,
Rohini
On Tue, May 10, 2016 at 7:27 AM, Harish Gopalan
<ha...@gmail.com> <ha...@gmail.com>
wrote:
Hi,
Is it possible to retrieve the original pig script from the MR job
configuration similar to hive i.e hive.query.string.
I tried to deserialize pig.pigContext value from the MR job configuration
but couldn't get it ?
Please advise.
Regards
Harish
Re: Retrieving Pig script from MR job config
Posted by Rohini Palaniswamy <ro...@gmail.com>.
You can find the pig script in pig.script setting. It is base64 encoded and
you will have to decode it. If the script is too long, it will be truncated
to 10K lines.
Regards,
Rohini
On Tue, May 10, 2016 at 7:27 AM, Harish Gopalan <ha...@gmail.com>
wrote:
> Hi,
>
> Is it possible to retrieve the original pig script from the MR job
> configuration similar to hive i.e hive.query.string.
>
> I tried to deserialize pig.pigContext value from the MR job configuration
> but couldn't get it ?
>
> Please advise.
>
> Regards
> Harish
>