You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Jeff Zhang (JIRA)" <ji...@apache.org> on 2009/07/16 16:54:15 UTC
[jira] Created: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Pig can not access reporter of PigHadoopLog in Load Func
--------------------------------------------------------
Key: PIG-889
URL: https://issues.apache.org/jira/browse/PIG-889
Project: Pig
Issue Type: Improvement
Components: impl
Affects Versions: 0.4.0
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Fix For: 0.4.0
I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735129#action_12735129 ]
Santhosh Srinivasan commented on PIG-889:
-----------------------------------------
The issue here is the lack of support for counters within Pig.
The intention of warn method in the PigLogger interface was to allow sources within Pig and UDFs for warning aggregation. Your use of the reporter within the logger is not supported. An implementation detail prevents the correct use of this interface for load functions. The Hadoop reporter object is provided in the getRecordReader, map and reduce calls. For load functions, Pig provides an interface and for UDFs, an abstract class. As a result, the logger instance cannot be initialized in the loaders till we decide to add a method to support it.
Will having the code from PigMapBase.map() in PigInputFormat.java.getRecordReader work for you?
{code}
PigHadoopLogger pigHadoopLogger = PigHadoopLogger.getInstance();
pigHadoopLogger.setAggregate(aggregateWarning);
pigHadoopLogger.setReporter(reporter);
PhysicalOperator.setPigLogger(pigHadoopLogger);
{code}
Note that this is a workaround for your situation. I would highly recommend that you move to the use of counters when they are supported.
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-889:
---------------------------
Status: Open (was: Patch Available)
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734119#action_12734119 ]
Jeff Zhang commented on PIG-889:
--------------------------------
Santhosh, Thank you for your review.
You are right, the testcase I provider will throw NullPointerException in local model.
After more investigation, I found that actually I can PigHadoopLogger.warn(Object o, String msg, Enum warningEnum) to increment Counter.
But it seems this method has some problems, because the Counter it generates is always one less than real Counter. And I found
this is because when Pig read the first record the reporter is null, but when it reads the second one, the reporter is not null. I guess the reason is that it is been set in PigMapBase (line 217) when do map operation on the first record.
I do some changes on my patch, The load funct PigStorageUsingReporter I provide is not for local model, it is just to make sure in mapreduce mode, the reporter should not been null.
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-889:
---------------------------
Attachment: Pig_889_Patch.txt
attach the patch
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-889:
---------------------------
Attachment: (was: Pig_889_Patch.txt)
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736389#action_12736389 ]
Jeff Zhang commented on PIG-889:
--------------------------------
Santhosh, you are right. I agree with you. The original purpose of PigHadoopLogger is not for Counter. We should not let the PigHadoopLogger to implement Counter for future extensibility.
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733825#action_12733825 ]
Santhosh Srinivasan commented on PIG-889:
-----------------------------------------
Comments:
The reporter inside the logger is setup correctly in PigInputFormat for Hadoop. However the usage of the logger to retrieve the reporter and then increment counters is flawed for the following reasons:
1. In the test case, the new loader uses PigHadoopLogger directly. When the loader is used in local mode, the notion of Hadoop disappears and the reference to PigHadoopLogger is not usable (i.e., will result in a NullPointerException).
{code}
+ @Override
+ public Tuple getNext() throws IOException {
+ PigHadoopLogger.getInstance().getReporter().incrCounter(
+ MyCounter.TupleCounter, 1);
+ return super.getNext();
+ }
{code}
2. The loggers were meant for warning aggregations. Here, there is a case being made to expand the capabilities to allow user defined counter aggregations. If thats the case, then new methods have to be added to the PigLogger interface.
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732136#action_12732136 ]
Alan Gates commented on PIG-889:
--------------------------------
Jeff,
Could you include a unit test that has a load func that fetcher the logger. That way we'll be sure you fix fixes what you want.
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735865#action_12735865 ]
Hadoop QA commented on PIG-889:
-------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12414215/Pig_889_Patch.txt
against trunk revision 797290.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/142/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/142/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/142/console
This message is automatically generated.
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736201#action_12736201 ]
Santhosh Srinivasan commented on PIG-889:
-----------------------------------------
The PigHadoopLogger implements the PigLogger interface. The only supported method for this interface is warn(). Supporting counters as part of Pig will involve part of what you suggest. While your implementation extends PigHadoopLogger, it is not generic to support counters in Pig. Other load functions will have to use direct references to PigHadoopLogger which is not the correct way of accessing and updating counters. Pig needs to extend the load function interface (and store function interface?) to allow access to counters.
Summary: Pig needs to support counters and its a slightly bigger topic. Extending functionality of existing classes that are meant for a different reason will make support difficult in the future.
If you agree, we can mark this issue as invalid and open a new jira that will capture requirements for supporting counters in Pig?
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732049#action_12732049 ]
Hadoop QA commented on PIG-889:
-------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12413696/Pig_889_Patch.txt
against trunk revision 793750.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/129/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/129/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/129/console
This message is automatically generated.
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-889:
---------------------------
Attachment: (was: Pig_889_Patch.txt)
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-889:
---------------------------
Status: Patch Available (was: Open)
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-889:
---------------------------
Attachment: Pig_889_Patch.txt
I added two new method to PigHadoopLog to support Counter
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736414#action_12736414 ]
Jeff Zhang commented on PIG-889:
--------------------------------
Santhosh, BTW, since the PigHadoopLogger is not for Counter, why having the reporter object in it ?
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732732#action_12732732 ]
Olga Natkovich commented on PIG-889:
------------------------------------
I am reviewing this patch
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-889:
---------------------------
Attachment: Pig_889_Patch.txt
attach the patch including testcase.
This TestCase indicates this patch can allow you to use reporter to increment your Counter in LoadFunc. This TestCase will fail if you do not apply this patch.
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-889:
---------------------------
Status: Patch Available (was: Open)
submit the patch
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Santhosh Srinivasan resolved PIG-889.
-------------------------------------
Resolution: Won't Fix
Release Note: As per the discussion with Jeff, closing the bug as won't fix
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-889:
---------------------------
Status: Patch Available (was: Open)
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-889:
---------------------------
Attachment: Pig_889_Patch.txt
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736091#action_12736091 ]
Jeff Zhang commented on PIG-889:
--------------------------------
Santhosh, in my opinion if we want to support Counter in Pig, we only have to initialize the reporter in PigInputFormat. Because in map and reduce phase, we already have initialize the reporter. (PigMapReduce.java : Line 236 and PigMapBase: Line 217)
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-889:
---------------------------
Attachment: (was: Pig_889_Patch.txt)
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-889:
---------------------------
Status: Open (was: Patch Available)
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736990#action_12736990 ]
Santhosh Srinivasan commented on PIG-889:
-----------------------------------------
PigHadoopLogger implements the PigLogger interface. As part of the implementation it uses the Hadoop reporter for aggregating the warning messages.
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-889:
---------------------------
Status: Open (was: Patch Available)
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of
PigHadoopLog in Load Func
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732721#action_12732721 ]
Hadoop QA commented on PIG-889:
-------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12413822/Pig_889_Patch.txt
against trunk revision 794937.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/133/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/133/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/133/console
This message is automatically generated.
> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
> Key: PIG-889
> URL: https://issues.apache.org/jira/browse/PIG-889
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.4.0
>
> Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized.
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.