You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Jimmy Hu (JIRA)" <ji...@apache.org> on 2012/07/05 23:04:33 UTC
[jira] [Created] (HIVE-3234) getting the reporter in the
recordwriter
Jimmy Hu created HIVE-3234:
------------------------------
Summary: getting the reporter in the recordwriter
Key: HIVE-3234
URL: https://issues.apache.org/jira/browse/HIVE-3234
Project: Hive
Issue Type: Improvement
Components: Serializers/Deserializers
Affects Versions: 0.9.1
Environment: any
Reporter: Jimmy Hu
Fix For: 0.9.1
We would like to generate some custom statistics and report back to map/reduce later wen implement the
FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3234) getting the reporter in the
recordwriter
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497303#comment-13497303 ]
Phabricator commented on HIVE-3234:
-----------------------------------
ashutoshc has requested changes to the revision "HIVE-3234 [jira] getting the reporter in the recordwriter".
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:1421 This method is called from FileSinkOperator::jobCloseOp() and from RCFileMergeMapper::jobClose() both of which can get access to reporter. I think we should make changes to propagate Reporter all the way through, instead of passing null.
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java:311 Here also it seems that reporter can be propagated in call hierachy from different join operators. So, we should make those changes instead of passing null.
REVISION DETAIL
https://reviews.facebook.net/D6699
BRANCH
h-3234
To: JIRA, ashutoshc, omalley
> getting the reporter in the recordwriter
> ----------------------------------------
>
> Key: HIVE-3234
> URL: https://issues.apache.org/jira/browse/HIVE-3234
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.9.1
> Environment: any
> Reporter: Jimmy Hu
> Assignee: Owen O'Malley
> Labels: newbie
> Fix For: 0.9.1
>
> Attachments: HIVE-3234.D6699.1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later wen implement the
> FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
> For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3234) getting the reporter in the
recordwriter
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-3234:
------------------------------
Attachment: HIVE-3234.D6699.1.patch
omalley requested code review of "HIVE-3234 [jira] getting the reporter in the recordwriter".
Reviewers: JIRA
hive-3234. pass reporter down to outputfileformat
We would like to generate some custom statistics and report back to map/reduce later wen implement the
FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
TEST PLAN
Added a unit test that uses a custom input output format, which
ensures the Progressable is a non-null Reporter.
REVISION DETAIL
https://reviews.facebook.net/D6699
AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
ql/src/test/org/apache/hadoop/hive/ql/io/udf/Rot13InputFormat.java
ql/src/test/org/apache/hadoop/hive/ql/io/udf/Rot13OutputFormat.java
ql/src/test/queries/clientpositive/custom_input_output_format.q
ql/src/test/results/clientpositive/custom_input_output_format.q.out
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/15939/
To: JIRA, omalley
> getting the reporter in the recordwriter
> ----------------------------------------
>
> Key: HIVE-3234
> URL: https://issues.apache.org/jira/browse/HIVE-3234
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.9.1
> Environment: any
> Reporter: Jimmy Hu
> Assignee: Owen O'Malley
> Labels: newbie
> Fix For: 0.9.1
>
> Attachments: HIVE-3234.D6699.1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later wen implement the
> FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
> For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3234) getting the reporter in the
recordwriter
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505340#comment-13505340 ]
Hudson commented on HIVE-3234:
------------------------------
Integrated in Hive-trunk-h0.21 #1821 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1821/])
HIVE-3234 : getting the reporter in the recordwriter (Owen Omalley via Ashutosh Chauhan) (Revision 1414266)
Result = FAILURE
hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414266
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java
* /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/udf
* /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/udf/Rot13InputFormat.java
* /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/udf/Rot13OutputFormat.java
* /hive/trunk/ql/src/test/queries/clientpositive/custom_input_output_format.q
* /hive/trunk/ql/src/test/results/clientpositive/custom_input_output_format.q.out
> getting the reporter in the recordwriter
> ----------------------------------------
>
> Key: HIVE-3234
> URL: https://issues.apache.org/jira/browse/HIVE-3234
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.9.1
> Environment: any
> Reporter: Jimmy Hu
> Assignee: Owen O'Malley
> Labels: newbie
> Fix For: 0.10.0
>
> Attachments: HIVE-3234.D6699.1.patch, HIVE-3234.D6699.2.patch, HIVE-3234.D6987.1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later wen implement the
> FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
> For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3234) getting the reporter in the
recordwriter
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498390#comment-13498390 ]
Phabricator commented on HIVE-3234:
-----------------------------------
ashutoshc has accepted the revision "HIVE-3234 [jira] getting the reporter in the recordwriter".
Thanks Owen for incorporating changes. Looks good.
REVISION DETAIL
https://reviews.facebook.net/D6699
BRANCH
h-3234
To: JIRA, ashutoshc, omalley
> getting the reporter in the recordwriter
> ----------------------------------------
>
> Key: HIVE-3234
> URL: https://issues.apache.org/jira/browse/HIVE-3234
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.9.1
> Environment: any
> Reporter: Jimmy Hu
> Assignee: Owen O'Malley
> Labels: newbie
> Fix For: 0.9.1
>
> Attachments: HIVE-3234.D6699.1.patch, HIVE-3234.D6699.2.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later wen implement the
> FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
> For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3234) getting the reporter in the
recordwriter
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504406#comment-13504406 ]
Ashutosh Chauhan commented on HIVE-3234:
----------------------------------------
Looks good. It seems that patch contains another patch. Can you get rid of ivysettings.xml?
+1 I am running tests now, will commit if tests pass.
> getting the reporter in the recordwriter
> ----------------------------------------
>
> Key: HIVE-3234
> URL: https://issues.apache.org/jira/browse/HIVE-3234
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.9.1
> Environment: any
> Reporter: Jimmy Hu
> Assignee: Owen O'Malley
> Labels: newbie
> Fix For: 0.9.1
>
> Attachments: HIVE-3234.D6699.1.patch, HIVE-3234.D6699.2.patch, HIVE-3234.D6987.1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later wen implement the
> FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
> For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3234) getting the reporter in the
recordwriter
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HIVE-3234:
-----------------------------------
Status: Open (was: Patch Available)
TestCliDriver_join40.q and TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat failed. Owen, can you take a look.
> getting the reporter in the recordwriter
> ----------------------------------------
>
> Key: HIVE-3234
> URL: https://issues.apache.org/jira/browse/HIVE-3234
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.9.1
> Environment: any
> Reporter: Jimmy Hu
> Assignee: Owen O'Malley
> Labels: newbie
> Fix For: 0.9.1
>
> Attachments: HIVE-3234.D6699.1.patch, HIVE-3234.D6699.2.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later wen implement the
> FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
> For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3234) getting the reporter in the
recordwriter
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497302#comment-13497302 ]
Ashutosh Chauhan commented on HIVE-3234:
----------------------------------------
Hey Owen, I left some comments on phabricator.
> getting the reporter in the recordwriter
> ----------------------------------------
>
> Key: HIVE-3234
> URL: https://issues.apache.org/jira/browse/HIVE-3234
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.9.1
> Environment: any
> Reporter: Jimmy Hu
> Assignee: Owen O'Malley
> Labels: newbie
> Fix For: 0.9.1
>
> Attachments: HIVE-3234.D6699.1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later wen implement the
> FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
> For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3234) getting the reporter in the
recordwriter
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HIVE-3234:
--------------------------------
Status: Patch Available (was: Open)
This patch passes in the real mapreduce reporter as the progressable for getHiveReportWriter. OutputFormats should still protect themselves from null in the Progressable, but the FileSinkOperator passes a Reporter from the mapreduce job.
> getting the reporter in the recordwriter
> ----------------------------------------
>
> Key: HIVE-3234
> URL: https://issues.apache.org/jira/browse/HIVE-3234
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.9.1
> Environment: any
> Reporter: Jimmy Hu
> Assignee: Owen O'Malley
> Labels: newbie
> Fix For: 0.9.1
>
> Attachments: HIVE-3234.D6699.1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later wen implement the
> FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
> For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3234) getting the reporter in the
recordwriter
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498394#comment-13498394 ]
Ashutosh Chauhan commented on HIVE-3234:
----------------------------------------
+1 will commit if tests pass.
> getting the reporter in the recordwriter
> ----------------------------------------
>
> Key: HIVE-3234
> URL: https://issues.apache.org/jira/browse/HIVE-3234
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.9.1
> Environment: any
> Reporter: Jimmy Hu
> Assignee: Owen O'Malley
> Labels: newbie
> Fix For: 0.9.1
>
> Attachments: HIVE-3234.D6699.1.patch, HIVE-3234.D6699.2.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later wen implement the
> FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
> For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3234) getting the reporter in the
recordwriter
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-3234:
------------------------------
Attachment: HIVE-3234.D6699.2.patch
omalley updated the revision "HIVE-3234 [jira] getting the reporter in the recordwriter".
Reviewers: JIRA, ashutoshc
I've updated the patch based on Ashutosh's feedback. In particular, I've
pushed the Reporter through the RowContainer.
REVISION DETAIL
https://reviews.facebook.net/D6699
AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java
ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java
ql/src/test/org/apache/hadoop/hive/ql/io/udf/Rot13InputFormat.java
ql/src/test/org/apache/hadoop/hive/ql/io/udf/Rot13OutputFormat.java
ql/src/test/queries/clientpositive/custom_input_output_format.q
ql/src/test/results/clientpositive/custom_input_output_format.q.out
To: JIRA, ashutoshc, omalley
> getting the reporter in the recordwriter
> ----------------------------------------
>
> Key: HIVE-3234
> URL: https://issues.apache.org/jira/browse/HIVE-3234
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.9.1
> Environment: any
> Reporter: Jimmy Hu
> Assignee: Owen O'Malley
> Labels: newbie
> Fix For: 0.9.1
>
> Attachments: HIVE-3234.D6699.1.patch, HIVE-3234.D6699.2.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later wen implement the
> FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
> For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3234) getting the reporter in the
recordwriter
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-3234:
------------------------------
Attachment: HIVE-3234.D6987.1.patch
omalley requested code review of "HIVE-3234 [jira] getting the reporter in the recordwriter".
Reviewers: JIRA
HIVE-3736 : hive unit test case build failure. (Ashish Singh via Ashutosh Chauhan)
We would like to generate some custom statistics and report back to map/reduce later wen implement the
FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D6987
AFFECTED FILES
ivy/ivysettings.xml
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java
ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java
ql/src/test/org/apache/hadoop/hive/ql/io/udf/Rot13InputFormat.java
ql/src/test/org/apache/hadoop/hive/ql/io/udf/Rot13OutputFormat.java
ql/src/test/queries/clientpositive/custom_input_output_format.q
ql/src/test/results/clientpositive/custom_input_output_format.q.out
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/16461/
To: JIRA, omalley
> getting the reporter in the recordwriter
> ----------------------------------------
>
> Key: HIVE-3234
> URL: https://issues.apache.org/jira/browse/HIVE-3234
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.9.1
> Environment: any
> Reporter: Jimmy Hu
> Assignee: Owen O'Malley
> Labels: newbie
> Fix For: 0.9.1
>
> Attachments: HIVE-3234.D6699.1.patch, HIVE-3234.D6699.2.patch, HIVE-3234.D6987.1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later wen implement the
> FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
> For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-3234) getting the reporter in the
recordwriter
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan resolved HIVE-3234.
------------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.9.1)
0.10.0
Committed to trunk and 0.10. Thanks, Owen!
> getting the reporter in the recordwriter
> ----------------------------------------
>
> Key: HIVE-3234
> URL: https://issues.apache.org/jira/browse/HIVE-3234
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.9.1
> Environment: any
> Reporter: Jimmy Hu
> Assignee: Owen O'Malley
> Labels: newbie
> Fix For: 0.10.0
>
> Attachments: HIVE-3234.D6699.1.patch, HIVE-3234.D6699.2.patch, HIVE-3234.D6987.1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later wen implement the
> FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
> For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3234) getting the reporter in the
recordwriter
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley reassigned HIVE-3234:
-----------------------------------
Assignee: Owen O'Malley
> getting the reporter in the recordwriter
> ----------------------------------------
>
> Key: HIVE-3234
> URL: https://issues.apache.org/jira/browse/HIVE-3234
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.9.1
> Environment: any
> Reporter: Jimmy Hu
> Assignee: Owen O'Malley
> Labels: newbie
> Fix For: 0.9.1
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later wen implement the
> FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object.
> For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira