[DISCUSS] ATLAS-5399 Apache Nutch metadata model and CrawlDb/HostDB import bridge

[lewis john mcgibbney <le...@apache.org>]

Hi dev@,

I opened ATLAS-5399 to catalog Apache Nutch crawls in Atlas without
creating one entity per URL.

JIRA: https://issues.apache.org/jira/browse/ATLAS-5399
PR:   https://github.com/apache/atlas/pull/748

What this adds
--------------

   - Nutch types in addons/models/7000-Nutch (crawl, seedlist, segment,
   domain, host, nutch_index_process). Fetch metrics live on
   nutch_crawl_hosts. Hosts are global per cluster: {hostname}@
   {clusterName}.
   - A batch import bridge (addons/nutch-bridge). Hosts come from Nutch
   HostDB when present, otherwise CrawlDb rollup. Seeds and segments are
   imported from the crawl directories.
   - Distro nutch-hook package and hook-bin/import-nutch.sh
   - Docs: docs/src/documents/Hook/HookNutch.md

Indexing lineage is a split with Nutch. This PR defines nutch_index_process
and the generic index DataSet shape; NUTCH-3210 (Nutch indexer-atlas) will
emit one process per IndexingJob. The two sides share atlas.cluster.name
and crawlId.

Why this shape
--------------
A large crawl would explode Atlas if every URL were an entity. Cataloging
crawls, segments, domains, and hosts keeps the graph bounded and still
gives lineage from crawl/segment to index process to search collection.

How to try it
-------------
Build Atlas, start the docker stack (postgres backend is enough), then:

  hook-bin/import-nutch.sh -c <crawlId> -d <crawldb> \
    [-H <hostdb>] [-s <seedDir>] [-g <segmentsDir>]

I ran this against a live docker Atlas with seed https://nutch.apache.org/
and confirmed crawl/seed/segment/domain/host entities plus lineage once the
Nutch IndexWriter ran. Screenshot is on the PR.

Reviews and comments on the JIRA or PR are welcome. A companion Nutch
change is NUTCH-3210. I will submit that PR soon after some refinements.

Thanks,
lewismc

http://people.apache.org/keys/committer/lewismc