You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org> on 2012/11/03 00:16:12 UTC
[jira] [Created] (SQOOP-678) Add counters handling to map reduce
submission engine
Jarek Jarcec Cecho created SQOOP-678:
----------------------------------------
Summary: Add counters handling to map reduce submission engine
Key: SQOOP-678
URL: https://issues.apache.org/jira/browse/SQOOP-678
Project: Sqoop
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Jarek Jarcec Cecho
Fix For: 2.0.0
We should implement counter handling in map reduce submission engine.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-678) Add counters handling to map reduce
submission engine
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492782#comment-13492782 ]
Hari Shreedharan commented on SQOOP-678:
----------------------------------------
>From what I see here, this basically is meant to expose the Map Reduce counters through the stats() method in the submission engine(through the Sqoop counters, of course)? I am planning to take a look at this one. For now, I am just planning to hard code a set of stats to be pulled and then later make it configurable. How does that sound?
> Add counters handling to map reduce submission engine
> -----------------------------------------------------
>
> Key: SQOOP-678
> URL: https://issues.apache.org/jira/browse/SQOOP-678
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 2.0.0
> Reporter: Jarek Jarcec Cecho
> Fix For: 2.0.0
>
>
> We should implement counter handling in map reduce submission engine.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-678) Add counters handling to map reduce
submission engine
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492867#comment-13492867 ]
Jarek Jarcec Cecho commented on SQOOP-678:
------------------------------------------
Yes, I believe that the counters make sense only after the submission successfully finish. I wouldn't worry about serialization in metastore itself -- there is already another JIRA (SQOOP-667) that is currently assigned to me that should cover that.
Jarcec
> Add counters handling to map reduce submission engine
> -----------------------------------------------------
>
> Key: SQOOP-678
> URL: https://issues.apache.org/jira/browse/SQOOP-678
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 2.0.0
> Reporter: Jarek Jarcec Cecho
> Fix For: 2.0.0
>
>
> We should implement counter handling in map reduce submission engine.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-678) Add counters handling to map reduce
submission engine
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492864#comment-13492864 ]
Hari Shreedharan commented on SQOOP-678:
----------------------------------------
You mean serialize and store them after job completion, right?
> Add counters handling to map reduce submission engine
> -----------------------------------------------------
>
> Key: SQOOP-678
> URL: https://issues.apache.org/jira/browse/SQOOP-678
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 2.0.0
> Reporter: Jarek Jarcec Cecho
> Fix For: 2.0.0
>
>
> We should implement counter handling in map reduce submission engine.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-678) Add counters handling to map reduce
submission engine
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492821#comment-13492821 ]
Jarek Jarcec Cecho commented on SQOOP-678:
------------------------------------------
Hi Hary,
thank you very much for looking into this issue. I've just linked it with SQOOP-671 that I believe needs to be solved first.
Otherwise you're right. I believe that we need to take important counters and propagate them throw Sqoop counters to framework (and further serialize them in Sqoop's metastore). I think that we do not need to transfer all mapreduce counters. I'm thinking that starting with "Number of transferred items" and "Number of transferred bytes" should be reasonable at the moment.
Jarcec
> Add counters handling to map reduce submission engine
> -----------------------------------------------------
>
> Key: SQOOP-678
> URL: https://issues.apache.org/jira/browse/SQOOP-678
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 2.0.0
> Reporter: Jarek Jarcec Cecho
> Fix For: 2.0.0
>
>
> We should implement counter handling in map reduce submission engine.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira