How to share Same Counter in Multiple Jobs?

[Savannah Beckett <sa...@yahoo.com>]

Hi,
  I chain multiple jobs in my program.  Job 1's reduce function has a counter.  
I want job 3's reduce function to read this Job 1's counter.  How?  

Thanks.


      

Re: How to share Same Counter in Multiple Jobs?

[Ted Yu <yu...@gmail.com>]

I wrote the following code today. We have our own flow execution logic which
calls the following to collect counters.

    enum COUNT_COLLECTION {
        LOG,            // log the counters
        ADD_TO_CONF        // add counters to JobConf
    }

    protected static void collectCounters(RunningJob running, JobConf
jobConf, EnumSet<COUNT_COLLECTION> collFlags) {
        try {
            Counters counters = running.getCounters();
            Collection<String> counterGroupNames = counters.getGroupNames();
            if (counterGroupNames == null) {
                LOG.warn("No counters returned from job " +
running.getJobName());
            } else {
                String[] groupsToCollect = { "Map-Reduce Framework",
"FileSystemCounters" };
                for (String counterGroupName : groupsToCollect) {
                    for (Iterator<Counter> iterator =
counters.getGroup(counterGroupName).iterator(); iterator.hasNext();) {
                        Counter counter = iterator.next();
                        String counterName =
counters.getGroup(counterGroupName).getDisplayName()+"."+
                        counter.getDisplayName();
                        if (collFlags.contains(COUNT_COLLECTION.LOG)) {
                            LOG.info(counterName + ": " +
counter.getCounter());
                        }
                    }
                }
            }
        } catch (IOException e) {
            LOG.error("unable to retrieve counters", e);
        }
    }

You can pass the counter from Job 1 to Job 3 via JobConf.

On Thu, Dec 9, 2010 at 9:45 PM, Savannah Beckett <
savannah_beckett30@yahoo.com> wrote:

> Hi,
>   I chain multiple jobs in my program.  Job 1's reduce function has a
> counter.  I want job 3's reduce function to read this Job 1's counter.
> How?
> Thanks.
>
>