You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/03/02 16:13:00 UTC
[jira] [Commented] (ATLAS-3641) Import Service: Support New
'zipDirect' Format for Import
[ https://issues.apache.org/jira/browse/ATLAS-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17049364#comment-17049364 ]
ASF subversion and git services commented on ATLAS-3641:
--------------------------------------------------------
Commit 6a49d94faf9613d3b10c33373500ca5e09b8dc7f in atlas's branch refs/heads/master from Ashutosh Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=6a49d94 ]
ATLAS-3641: Import Service: Support zipDirect format of import. Test files
> Import Service: Support New 'zipDirect' Format for Import
> ---------------------------------------------------------
>
> Key: ATLAS-3641
> URL: https://issues.apache.org/jira/browse/ATLAS-3641
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
> Affects Versions: trunk
> Reporter: Ashutosh Mestry
> Assignee: Ashutosh Mestry
> Priority: Major
> Fix For: trunk
>
> Attachments: ATLAS-3641-Import-Service-Support-zipDirect-format-o.patch
>
>
> *Background*
> Existing exported ZIP formats stores the entities and the order of import in 2 separate files. During import it is thus necessary for both these (entities and order) to be in memory to be able to correctly import.
> This causes large import files not function well. Also it puts a lot of pressure on memory.
> Existing format also uses 1 zip entry per entity. This does not effectively compress the files.
> The new format addresses these short comings.
> *Solution*
> Create a format that has:
> * Entities are stored in the order such that they are correctly imported.
> * Lends itself to be able to read as a stream. This will reduce memory constraints.
> * Store entities as 1 zip entry. This will improve compression rate.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)