[jira] [Created] (ATLAS-4922) Atlas Parallel Import Functionality using Kafka

["Disha Talreja (Jira)" <ji...@apache.org>]

Disha Talreja created ATLAS-4922:
------------------------------------

             Summary: Atlas Parallel Import Functionality using Kafka
                 Key: ATLAS-4922
                 URL: https://issues.apache.org/jira/browse/ATLAS-4922
             Project: Atlas
          Issue Type: New Feature
          Components:  atlas-core
            Reporter: Disha Talreja
            Assignee: Disha Talreja


Develop an import process flow in Apache Atlas that leverages Kafka to manage parallel import requests. This solution will use a dedicated Kafka topic (ATLAS_IMPORT) to ensure import requests are processed efficiently and without conflicts. The ImportService will be enhanced to consume and process each message from the topic, enabling seamless behind-the-scenes import execution.

Users will be able to submit new import requests after the staging process, which prepares the file stream for Kafka, is complete. This approach supports concurrent imports and enhances the handling of import operations in Apache Atlas.

Additionally, this feature will introduce two new API endpoints for tracking the current status of imports. These endpoints will allow users to check both the overall progress and detailed status of each import request at any stage of the import lifecycle (e.g., "PROCESSING," "PROCESSED," "COMPLETE") using the unique request ID. Users will also gain access to request metadata, providing insights into the status and progress of their imports.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)