You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by "Shashank Gupta (JIRA)" <ji...@apache.org> on 2014/03/06 06:37:42 UTC
[jira] [Comment Edited] (JCR-3731) Multi-threaded migration of
binary files from FileSystem to S3 datastore
[ https://issues.apache.org/jira/browse/JCR-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13922040#comment-13922040 ]
Shashank Gupta edited comment on JCR-3731 at 3/6/14 5:36 AM:
-------------------------------------------------------------
Attached patched.
Created thread pool to migrate files from file system to S3. For
570 MB initial data, time reduced to 70s from 307s.
Result:
h5. Migration using 1 thread:
{code}
18.12.2013 05:08:36.877 *INFO* [FelixStartLevel]
org.apache.jackrabbit.core.data.CachingDataStore Uploaded: {3875}/{3875} files,
{570535403}/{570535403} size data, time taken {307} sec
{code}
h5. Migration using 10 threads:
{code}
18.12.2013 04:56:52.176 *INFO* [FelixStartLevel]
org.apache.jackrabbit.core.data.CachingDataStore Uploaded: {3875}/{3875} files,
{570535403}/{570535403} size data, time taken {70} sec
{code}
was (Author: shgupta):
Attached patched.
Created thread pool to migrate files from file system to S3. For
570 MB initial data, time reduced to 70s from 307s.
Result:
h5. Migration using 1 thread:
{code}
18.12.2013 05:08:36.877 *INFO* [FelixStartLevel]
com.day.crx.cloud.s3.ds.CachingDataStore Uploaded: {3875}/{3875} files,
{570535403}/{570535403} size data, time taken {307} sec
{code}
> Multi-threaded migration of binary files from FileSystem to S3 datastore
> -------------------------------------------------------------------------
>
> Key: JCR-3731
> URL: https://issues.apache.org/jira/browse/JCR-3731
> Project: Jackrabbit Content Repository
> Issue Type: Sub-task
> Components: jackrabbit-core
> Affects Versions: 2.7.4
> Reporter: Shashank Gupta
> Fix For: 2.7.5
>
> Attachments: JCR-3651-svn-updated.patch
>
>
> As per today, when we switch repository from FileDataStore to S3DataStore all binary files are migrated from local file system to S3Datastore. As per today this process is single threaded and takes lot of time. For e.g. for 1G intial content, it takes around 5 min to migrated from ec2 instance to S3.
> It can be made faster by migrating content in multi-threaded environment.
--
This message was sent by Atlassian JIRA
(v6.2#6252)