You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Prashant Kommireddi <pr...@gmail.com> on 2011/12/09 08:28:32 UTC

Aggregate Warning does not work as expected on Embedding Pig in Java 0.9.1

I noticed the property "aggregate.warning" is not being set by default when
running PigServer, embedding Pig in Java.

I was initially creating a PigServer object this way:
 PigServer pigServer = new PigServer(ExecType.MAPREDUCE);

But this generated detailed logs in the log directory, I noticed this only
when the DNs ran out of disk space. I had to code around this to make it
work:

Properties properties = PropertiesUtil.loadDefaultProperties();
properties.setProperty("aggregate.warning", "true");
PigServer pigServer = new PigServer(ExecType.MAPREDUCE, properties);

The behavior between Pig scripting and Embedded Pig be similar? Looking at
the main constructor of PigServer, it looks like "aggregateWarning" is set
to false if its not loaded in to Properties object.

public PigServer(PigContext context, boolean connect) throws ExecException {
        this.pigContext = context;
        currDAG = new Graph(false);

        aggregateWarning =
"true".equalsIgnoreCase(pigContext.getProperties().getProperty("aggregate.warning"));
        isMultiQuery =
"true".equalsIgnoreCase(pigContext.getProperties().getProperty("opt.multiquery","true"));

        jobName = pigContext.getProperties().getProperty(
                PigContext.JOB_NAME,
                PigContext.JOB_NAME_PREFIX + ":DefaultJobName");

        if (connect) {
            pigContext.connect();
        }

        addJarsFromProperties();
    }

Looks like a bug?

Re: Aggregate Warning does not work as expected on Embedding Pig in Java 0.9.1

Posted by Prashant Kommireddi <pr...@gmail.com>.
Hi Thejas,

I have opened Jira for this https://issues.apache.org/jira/browse/PIG-2425

Thanks,
Prashant

On Mon, Dec 12, 2011 at 12:34 PM, Thejas Nair <th...@hortonworks.com>wrote:

> Hi Prashant,
> I think it does make sense to enable warning aggregation even used through
> PigServer.
> Can you please open a jira ? (and a patch if possible !)
>
> Thanks,
> Thejas
>
>
>
> On 12/8/11 11:28 PM, Prashant Kommireddi wrote:
>
>> I noticed the property "aggregate.warning" is not being set by default
>> when
>> running PigServer, embedding Pig in Java.
>>
>> I was initially creating a PigServer object this way:
>>  PigServer pigServer = new PigServer(ExecType.MAPREDUCE);
>>
>> But this generated detailed logs in the log directory, I noticed this only
>> when the DNs ran out of disk space. I had to code around this to make it
>> work:
>>
>> Properties properties = PropertiesUtil.**loadDefaultProperties();
>> properties.setProperty("**aggregate.warning", "true");
>> PigServer pigServer = new PigServer(ExecType.MAPREDUCE, properties);
>>
>> The behavior between Pig scripting and Embedded Pig be similar? Looking at
>> the main constructor of PigServer, it looks like "aggregateWarning" is set
>> to false if its not loaded in to Properties object.
>>
>> public PigServer(PigContext context, boolean connect) throws
>> ExecException {
>>         this.pigContext = context;
>>         currDAG = new Graph(false);
>>
>>         aggregateWarning =
>> "true".equalsIgnoreCase(**pigContext.getProperties().**
>> getProperty("aggregate.**warning"));
>>         isMultiQuery =
>> "true".equalsIgnoreCase(**pigContext.getProperties().**
>> getProperty("opt.multiquery","**true"));
>>
>>         jobName = pigContext.getProperties().**getProperty(
>>                 PigContext.JOB_NAME,
>>                 PigContext.JOB_NAME_PREFIX + ":DefaultJobName");
>>
>>         if (connect) {
>>             pigContext.connect();
>>         }
>>
>>         addJarsFromProperties();
>>     }
>>
>> Looks like a bug?
>>
>>
>

Re: Aggregate Warning does not work as expected on Embedding Pig in Java 0.9.1

Posted by Thejas Nair <th...@hortonworks.com>.
Hi Prashant,
I think it does make sense to enable warning aggregation even used 
through PigServer.
Can you please open a jira ? (and a patch if possible !)

Thanks,
Thejas


On 12/8/11 11:28 PM, Prashant Kommireddi wrote:
> I noticed the property "aggregate.warning" is not being set by default when
> running PigServer, embedding Pig in Java.
>
> I was initially creating a PigServer object this way:
>   PigServer pigServer = new PigServer(ExecType.MAPREDUCE);
>
> But this generated detailed logs in the log directory, I noticed this only
> when the DNs ran out of disk space. I had to code around this to make it
> work:
>
> Properties properties = PropertiesUtil.loadDefaultProperties();
> properties.setProperty("aggregate.warning", "true");
> PigServer pigServer = new PigServer(ExecType.MAPREDUCE, properties);
>
> The behavior between Pig scripting and Embedded Pig be similar? Looking at
> the main constructor of PigServer, it looks like "aggregateWarning" is set
> to false if its not loaded in to Properties object.
>
> public PigServer(PigContext context, boolean connect) throws ExecException {
>          this.pigContext = context;
>          currDAG = new Graph(false);
>
>          aggregateWarning =
> "true".equalsIgnoreCase(pigContext.getProperties().getProperty("aggregate.warning"));
>          isMultiQuery =
> "true".equalsIgnoreCase(pigContext.getProperties().getProperty("opt.multiquery","true"));
>
>          jobName = pigContext.getProperties().getProperty(
>                  PigContext.JOB_NAME,
>                  PigContext.JOB_NAME_PREFIX + ":DefaultJobName");
>
>          if (connect) {
>              pigContext.connect();
>          }
>
>          addJarsFromProperties();
>      }
>
> Looks like a bug?
>