You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Lewis John McGibbney (Jira)" <ji...@apache.org> on 2025/12/18 03:31:00 UTC
[jira] [Commented] (NUTCH-3131) Nutch Metrics Refactoring & Enhancements
[ https://issues.apache.org/jira/browse/NUTCH-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18046082#comment-18046082 ]
Lewis John McGibbney commented on NUTCH-3131:
---------------------------------------------
I plan to add more to this issue but I won't be able to get around to it for a little while.
> Nutch Metrics Refactoring & Enhancements
> ----------------------------------------
>
> Key: NUTCH-3131
> URL: https://issues.apache.org/jira/browse/NUTCH-3131
> Project: Nutch
> Issue Type: Improvement
> Components: metrics
> Reporter: Lewis John McGibbney
> Assignee: Lewis John McGibbney
> Priority: Major
> Attachments: nutch_metrics_report.pdf
>
>
> The recent “Hadoop Metrics Analysis and Improvement Suggestions” report (attached PDF) identified 88 counter increment operations across 17 files with multiple anti-patterns and missing key observability signals (inconsistent naming, repeated counter lookups, missing latency/error context, cardinality risks, etc.).
> This ticket covers the implementation of all high and medium-priority improvements from the report (Phases 1–3). Phase 4 (external export, dashboard, testing) will be handled in separate tickets.
> Acceptance criteria for consideration
> * All Hadoop counter group and counter names are defined in a single source of truth (new class NutchMetricConstants or NutchMetrics).
> * No hardcoded counter group/name strings remain in the 17 affected files.
> * All frequently used counters (especially in hot paths – Fetcher, FetcherThread, Generator, Parser, Indexer) are cached in instance variables during setup(Context) / setup() and reused.
> * Latency metrics (fetch, parse, index) are added with proper timing and recorded via Hadoop counters (average + count).
> * Error counters include error type/context where feasible (at least class-level granularity).
> * Counter naming is fully standardized (camelCase counters, PascalCase groups).
> * A lightweight MetricsHelper utility class exists and is used across components.
> * Thread-safe accumulation (AtomicLong/TDigest) is consolidated via ThreadSafeMetrics or equivalent and flushed correctly to Hadoop counters in cleanup().
> * Resource utilization metrics (queue sizes, depths) are added for Fetcher.
> * Basic metrics validation is executed at the end of each job (warn on impossible conditions).
> * No regression in existing counter values (verified via existing integration tests or new sanity job).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)