You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Ashutosh Mestry (Jira)" <ji...@apache.org> on 2019/09/10 21:29:00 UTC
[jira] [Comment Edited] (ATLAS-3396) ZipSource: Usage of In-memory
Maps Limits the Size of Imports Performed
[ https://issues.apache.org/jira/browse/ATLAS-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926904#comment-16926904 ]
Ashutosh Mestry edited comment on ATLAS-3396 at 9/10/19 9:28 PM:
-----------------------------------------------------------------
Cherry pick to master attached.
Successful master PC build: [https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1399/]
was (Author: ashutoshm):
Master PC build: [https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1399/]
> ZipSource: Usage of In-memory Maps Limits the Size of Imports Performed
> -----------------------------------------------------------------------
>
> Key: ATLAS-3396
> URL: https://issues.apache.org/jira/browse/ATLAS-3396
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Affects Versions: 0.8.4
> Reporter: Ashutosh Mestry
> Assignee: Ashutosh Mestry
> Priority: Major
> Fix For: 0.8.4, trunk
>
> Attachments: ATLAS-3396-ZipSource-Memory-usage-improvement-for-la-branch-0.8.patch, ATLAS-3396-ZipSourceWithBackingDirectory-Implementat-master.patch
>
>
> *Background*
> _ZipSource_ is the container for working with Atlas-exported ZIP files. The current implementation fetches the contents of the entire ZIP file in memory. This greatly limits the size of ZIP files that can be imported.
> *Solution Guidance*
> New class _ZipSourceWIthBackingDirectory_, this will use a user-specified (or default) temporary directory to preserve the contents of the ZIP file. This will reduce the memory burden and large ZIP files can be imported.
> Minor refactoring to _ImportService_ will be necessary to support the new variation of _EntityImportStream._
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)