[jira] [Updated] (SLING-2707) Support of chunked file upload into Sling

["Shashank Gupta (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SLING-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shashank Gupta updated SLING-2707:
----------------------------------

    Attachment: uploadclient.jar

> What was used as server/endpoint for the above comparison? 
CQ 5.6 server with S3 datastore. 

I have implemented a servlet endpoint on amazon ec2 instance which can accept parallel file chunks and merge them. Also attached upload client which can upload chunks in parallel.   

Environment: Amazon ec2 instance 
OS: RHEL-6.3
Server: CQ 5.6 Load 20
CRX datastore: file system
Server endpoint: http://ec2-23-20-237-244.compute-1.amazonaws.com:4502
Usage: java -DparallelThreads=<number of parallel connections> -DbaseUrl=http://ec2-23-20-237-244.compute-1.amazonaws.com:4502 -DfilePath=<file-path> -jar uploadclient.jar
if number of parallel connections = 1 : all chunks uploaded in serial manner
file-path = path of file which is being uploaded.  file is uploaded into /content/dam/<file_name>

Results: 
client : my laptop (my org network)
File size : 54 MB
Chunk size : 5 MB
Number of chunks : 11
Time taken in serial upload : 1735 sec ~= 29 minutes
Time taken in parallel upload with 6 parallel connections: 691 sec ~=  11 minutes

There is clear advantage of faster upload using parallel connections.  

PS: Since crx datastore detects duplicate binary, please change binary a bit so that their SHA-1 hash is different. 
                
> Support of chunked file upload into Sling
> -----------------------------------------
>
>                 Key: SLING-2707
>                 URL: https://issues.apache.org/jira/browse/SLING-2707
>             Project: Sling
>          Issue Type: New Feature
>          Components: General
>            Reporter: Shashank Gupta
>         Attachments: uploadclient.jar
>
>
> Use cases: 
> 1. Large file upload - With high speed internet connections, advent of cloud and HD going mainstream, Sling should support large files (> 2GB) upload.
> 2. Fault tolerant uploads - Sling should provide capability to resume upload from failure point. It should not require client to restart the complete upload process. 
> 3. Faster upload: Sling should support its clients to initiate multiple connection and upload file chunks in parallel. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira