[jira] [Created] (NUTCH-3149) Investigate Remote Shuffle Service Integration (Apache Uniffle / Celeborn) for Shuffle-Intensive Nutch Jobs

["Lewis John McGibbney (Jira)" <ji...@apache.org>]

Lewis John McGibbney created NUTCH-3149:
-------------------------------------------

             Summary: Investigate Remote Shuffle Service Integration (Apache Uniffle / Celeborn) for Shuffle-Intensive Nutch Jobs
                 Key: NUTCH-3149
                 URL: https://issues.apache.org/jira/browse/NUTCH-3149
             Project: Nutch
          Issue Type: Improvement
          Components: shuffle
            Reporter: Lewis John McGibbney
            Assignee: Lewis John McGibbney


Several core jobs are shuffle-intensive, meaning they spend significant time and resources moving intermediate data between map and reduce phases. Remote shuffle services (such as Apache Uniffle and Apache Celeborn) offload this work to dedicated nodes, potentially improving:
 * Job execution time
 * Fault tolerance
 * Resource utilization
 * Scalability for large crawls

h4. Shuffle-Intensive Nutch Jobs Identified
||Job||Shuffle Intensity||Key Characteristics||
|LinkRank|Extreme|Iterative (10+ iterations), invert + analyze per iteration|
|WebGraph|Very High|3 sequential jobs (OutlinkDb → InlinkDb → NodeDb)|
|LinkDb|High|Full link inversion across segments|
|CrawlDb|High|URL grouping from multiple segments|
|Generator|Medium-High|Score-based selection + host/domain partitioning|
|Indexer|Medium-High|Multi-source aggregation (CrawlDb, LinkDb, segments)|
|Deduplication|Medium|Content signature-based grouping|
|Fetcher|Low|Primarily map-only, minimal shuffle|
h4. Key Metrics for Measuring Shuffle Intensity

Shuffle intensity can be determined from Hadoop's built-in TaskCounter and FileSystemCounter groups:
 * MAP_OUTPUT_BYTES / HDFS_BYTES_READ → Shuffle Ratio

 * SPILLED_RECORDS / MAP_OUTPUT_RECORDS → Spill Ratio

 * FILE_BYTES_READ + FILE_BYTES_WRITTEN → Local disk I/O overhead

Jobs with Shuffle Ratio > 2.0x or Spill Ratio > 2.0x are strong candidates for remote shuffle optimization.

An initial step would be to implement a configurable (off/off switch) reporting capability to write shuffle reports to the Nutch log. This will enable Nutch admins to determine over time if a remote shuffle candidate would benefit their Nutch jobs.

A future ticket will establish an experiment (comparative analysis) for Nutch shuffle data between Hadoop native (no changes), Uniffle and Celeborn.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)