[jira] [Created] (ATLAS-3641) Import Service: Support New 'zipDirect' Format for Import

["Ashutosh Mestry (Jira)" <ji...@apache.org>]

Ashutosh Mestry created ATLAS-3641:
--------------------------------------

             Summary: Import Service: Support New 'zipDirect' Format for Import
                 Key: ATLAS-3641
                 URL: https://issues.apache.org/jira/browse/ATLAS-3641
             Project: Atlas
          Issue Type: Improvement
          Components:  atlas-core
    Affects Versions: trunk
            Reporter: Ashutosh Mestry
            Assignee: Ashutosh Mestry
             Fix For: trunk


*Background*

Existing exported ZIP formats stores the entities and the order of import in 2 separate files. During import it is thus necessary for both these (entities and order) to be in memory to be able to correctly import.

This causes large import files not function well. Also it puts a lot of pressure on memory.

Existing format also uses 1 zip entry per entity. This does not effectively compress the files.

 The new format addresses these short comings.

*Solution*

Create a format that has:
 * Entities are stored in the order such that they are correctly imported.
 * Lends itself to be able to read as a stream. This will reduce memory constraints.
 * Store entities as 1 zip entry. This will improve compression rate.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)