You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Lewis John McGibbney (Jira)" <ji...@apache.org> on 2026/09/04 05:56:00 UTC

[jira] [Updated] (NUTCH-3210) Add Atlas IndexWriter to emit crawl/segment lineage to Apache Atlas

     [ https://issues.apache.org/jira/browse/NUTCH-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lewis John McGibbney updated NUTCH-3210:
----------------------------------------
    Description: 
Add an Atlas IndexWriter plugin that, at IndexingJob time, upserts lineage into [Apache Atlas|http://atlas.apache.org/] via AtlasClientV2 REST.

*Motivation*

Atlas will own Nutch typedefs and a CrawlDb batch bridge (ATLAS-XXXX). Index-time lineage (crawl/segment to index DataSet) belongs in Nutch so it runs with the existing indexer plugin path.

*Scope*
 # New plugin src/plugin/indexer-atlas implementing IndexWriter
 # Configure via conf/index-writers.xml and plugin.includes
 # One nutch_index_process per IndexingJob (not per URL)
 # Process.inputs: nutch_crawl and/or nutch_segment (must already exist in Atlas)
 # Process.outputs: generic DataSet named by collection/index + clusterName
 # Optional updates to existing crawl–host relationship indexedCount / lastIndexedTime only when the host already exists
 # Auth: Atlas REST username/password or token; atlas.cluster.name shared with the Atlas bridge
 # Unit tests with a mocked Atlas client; docs for config keys

*Out of scope*
 # Reusing indexer-kafka (payload is NutchDocument JSON, not HookNotification)
 # AtlasHook / Kafka ATLAS_HOOK
 # Creating crawl, seedlist, domain, or host entities (Atlas CrawlDb bridge)
 # URL-level Atlas entities
 # Restoring NutchServer / JobManager metadata

Depends on: ATLAS-5399 (Nutch type model; process/index DataSet types must exist)

  was:
Add an Atlas IndexWriter plugin that, at IndexingJob time, upserts lineage into [Apache Atlas|http://atlas.apache.org] via AtlasClientV2 REST.

*Motivation*

Atlas will own Nutch typedefs and a CrawlDb batch bridge (ATLAS-XXXX). Index-time lineage (crawl/segment to index DataSet) belongs in Nutch so it runs with the existing indexer plugin path.

*Scope*
 # New plugin src/plugin/indexer-atlas implementing IndexWriter
 # Configure via conf/index-writers.xml and plugin.includes
 # One nutch_index_process per IndexingJob (not per URL)
 # Process.inputs: nutch_crawl and/or nutch_segment (must already exist in Atlas)
 # Process.outputs: generic DataSet named by collection/index + clusterName
 # Optional updates to existing crawl–host relationship indexedCount / lastIndexedTime only when the host already exists
 # Auth: Atlas REST username/password or token; atlas.cluster.name shared with the Atlas bridge
 # Unit tests with a mocked Atlas client; docs for config keys

*Out of scope*
 # Reusing indexer-kafka (payload is NutchDocument JSON, not HookNotification)
 # AtlasHook / Kafka ATLAS_HOOK
 # Creating crawl, seedlist, domain, or host entities (Atlas CrawlDb bridge)
 # URL-level Atlas entities
 # Restoring NutchServer / JobManager metadata

Depends on: ATLAS-XXXX (Nutch type model; process/index DataSet types must exist)


> Add Atlas IndexWriter to emit crawl/segment lineage to Apache Atlas
> -------------------------------------------------------------------
>
>                 Key: NUTCH-3210
>                 URL: https://issues.apache.org/jira/browse/NUTCH-3210
>             Project: Nutch
>          Issue Type: New Feature
>          Components: indexer, plugin
>    Affects Versions: 1.24
>            Reporter: Lewis John McGibbney
>            Assignee: Lewis John McGibbney
>            Priority: Major
>             Fix For: 1.24
>
>
> Add an Atlas IndexWriter plugin that, at IndexingJob time, upserts lineage into [Apache Atlas|http://atlas.apache.org/] via AtlasClientV2 REST.
> *Motivation*
> Atlas will own Nutch typedefs and a CrawlDb batch bridge (ATLAS-XXXX). Index-time lineage (crawl/segment to index DataSet) belongs in Nutch so it runs with the existing indexer plugin path.
> *Scope*
>  # New plugin src/plugin/indexer-atlas implementing IndexWriter
>  # Configure via conf/index-writers.xml and plugin.includes
>  # One nutch_index_process per IndexingJob (not per URL)
>  # Process.inputs: nutch_crawl and/or nutch_segment (must already exist in Atlas)
>  # Process.outputs: generic DataSet named by collection/index + clusterName
>  # Optional updates to existing crawl–host relationship indexedCount / lastIndexedTime only when the host already exists
>  # Auth: Atlas REST username/password or token; atlas.cluster.name shared with the Atlas bridge
>  # Unit tests with a mocked Atlas client; docs for config keys
> *Out of scope*
>  # Reusing indexer-kafka (payload is NutchDocument JSON, not HookNotification)
>  # AtlasHook / Kafka ATLAS_HOOK
>  # Creating crawl, seedlist, domain, or host entities (Atlas CrawlDb bridge)
>  # URL-level Atlas entities
>  # Restoring NutchServer / JobManager metadata
> Depends on: ATLAS-5399 (Nutch type model; process/index DataSet types must exist)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)