You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Bill Graham (JIRA)" <ji...@apache.org> on 2012/11/08 08:26:12 UTC

[jira] [Commented] (PIG-2924) PigStats should not be assuming all Storage classes to be file-based storage

    [ https://issues.apache.org/jira/browse/PIG-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493019#comment-13493019 ] 

Bill Graham commented on PIG-2924:
----------------------------------

Sorry for the delay on the review Chelsoo. Looking good. A few more comments. Let me know what you think.

- I think we need to pass the {{POStore}} instead of the location. PigStorage impls provided by random parties might not all abide by a unique namespacing convention in their location syntax. For example, {{VerticaStorer}} uses a syntax like "{[db_schema].[table_name]}" (curly brackets included). Another implementor could use the same syntax.  
- JobStats.getOuputSize could be simplified by doing this, which is more commonly done:
{noformat}
String reporterNames = conf.get(
   PigStatsOutputSizeReader.OUTPUT_SIZE_READER_KEY,
   FileBasedOutputSizeReader.class.getCanonicalName());
{noformat}
- Does {{PigContext.instantiateFuncFromSpec(className)}} (without appending "()") not work?
- It seems like it would be reasonable for {{PigStatsOutputSizeReader.getOutputSize}} to throw IOException all the way up to {{JobStats}}.
- Let's make {{DummyOutputSizeReader}} an inner class of {{TestJobStats}} since that package is already totally bloated.
- In {{pig.properties}} reducers' should not have an apostrophe (no possessive for inanimate objects).
                
> PigStats should not be assuming all Storage classes to be file-based storage
> ----------------------------------------------------------------------------
>
>                 Key: PIG-2924
>                 URL: https://issues.apache.org/jira/browse/PIG-2924
>             Project: Pig
>          Issue Type: Bug
>          Components: tools
>    Affects Versions: 0.9.2, 0.10.0
>            Reporter: Harsh J
>            Assignee: Cheolsoo Park
>         Attachments: PIG-2924-2.patch, PIG-2924-3.patch, PIG-2924-4.patch, PIG-2924.patch
>
>
> Using PigStatsUtil (like Oozie does) to collect JobStats for jobs that use a HBaseStorage blows up when the stats are asked to be accumulated.
> This is because JobStats (which adds stuff up) is assuming all storages are file based and that it can do listStatus/etc. operations on their filespec-provided filename. For HBaseStorage, this is set to the tablename and there's no such file, leading to an exception (FileNotFound or Invalid URI - depending on using 'tablename' or 'hbase://tablename').

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira