You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Chetan Mehrotra <ch...@gmail.com> on 2014/02/18 08:25:10 UTC
Efficient import of binary data into Oak
Hi,
Currently in a Sling based application where a user uploads a file to
the JCR following sequence of steps are executed
1. User uploads file via HTTP request mostly using Multi-Part form
data based upload
2. Sling uses Commons File Upload to parse the multi-part request
which uses a DiskFileItemFactory and write the binary content to a
temporary file (for file size > 256 KB) [1]
3. Later the servlet would access the JCR Session and create a Binary
value by extracting the InputStream
4. The file content would then be spooled into the BlobStore
Effect of different blobstore
----------------------------------------
Now depending on the type of BlobStore one of the following code flow
would happen
A - JR2 DataStores - The inputstream would be copied to file
B - S3DataStore - The AWS SDK would be creating a temporary file and
then that file content would be streamed back to the S3
C - Segment - Content from InputStream would be stored as part of
various segments
D - MongoBlobStore - Content from InputStream would be pushed to
remote mongo via multiple remote calls
Things to note in above sequence
1. Uploaded content is copied twice.
2. The whole content is spooled via InputStream through JVM Heap
Possible areas of Improvement
--------------------------------
1. If the BlobStore is finally using some File (on same hard disk not
NFS) then it might be better to *move* the file which was created in
upload. This would help local FileDataStore and S3DataStore
2. Avoid spooling via InputStream if possible. Spooling via IS is slow
[3]. Though in most cases we use efficient buffered copy which is
marginally slower than NIO based variants. However avoiding moving
byte[] might reduce pressure on GC (probably!)
Changes required
------------------------
If we can have a way to create JCR Binary implementations which
enables DataStore/BlobStore to efficiently transfer content then that
would help.
For example for File based DS the Binary created can keep a reference
to the source File object and that Binary is used in JCR API.
Eventually the FileDataStore can treat it in a different way and move
the file.
Another example is S3DataStore - In some cases the file has already
been transferred to S3 using other options. And the user wants to
transfer the S3 file from its bucket to our bucket. So a Binary
implementation which can just wrap the S3 url would enable the
S3DataStore to transfer the content without streaming all content
again [4]
Any thoughts on the best way to enable users of Oak to create Binaries
via other means (compared to current mode which only enables via
InputStream) and enable the DataStores to make use of such binaries?
Chetan Mehrotra
[1]https://github.com/apache/sling/blob/trunk/bundles/engine/src/main/java/org/apache/sling/engine/impl/parameters/ParameterSupport.java#L190
[2] http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/PutObjectRequest.html
[3] http://www.baptiste-wicht.com/2010/08/file-copy-in-java-benchmark/3/
[4] http://stackoverflow.com/questions/9664904/best-way-to-move-files-between-s3-buckets
Re: Efficient import of binary data into Oak
Posted by Tobias Bocanegra <tr...@apache.org>.
hi,
In CRX we solved this problem with CRX providing the "blob factory",
i.e. CRX already creates the appropriate structures for exactly the
upload case, the multipart handler then just uses the blob for writing
the output, too. Unfortunately we never got this into the JCR spec
(IIRC).
Also, moving the temporary file is rarely an performance improvement,
as /tmp partitions are separate from the datastore ones. so the FS
needs to copy the file anyways.
regards, toby
On Tue, Feb 18, 2014 at 3:16 AM, Chetan Mehrotra
<ch...@gmail.com> wrote:
> On Tue, Feb 18, 2014 at 3:48 PM, Jukka Zitting <ju...@gmail.com> wrote:
>> Something like S3InputStream.getURL() should work just fine for that use case:
>
> That also would work with the caveat that in between layer do not
> decorate the InputStream in any form.
>
>
>
>
> Chetan Mehrotra
Re: Efficient import of binary data into Oak
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Feb 18, 2014 at 6:16 AM, Chetan Mehrotra
<ch...@gmail.com> wrote:
> That also would work with the caveat that in between layer do not
> decorate the InputStream in any form.
Right, good point. At the moment we don't decorate streams within Oak,
and I don't see any big reasons why we'd want to start doing that.
BR,
Jukka Zitting
Re: Efficient import of binary data into Oak
Posted by Chetan Mehrotra <ch...@gmail.com>.
On Tue, Feb 18, 2014 at 3:48 PM, Jukka Zitting <ju...@gmail.com> wrote:
> Something like S3InputStream.getURL() should work just fine for that use case:
That also would work with the caveat that in between layer do not
decorate the InputStream in any form.
Chetan Mehrotra
Re: Efficient import of binary data into Oak
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Feb 18, 2014 at 4:41 AM, Chetan Mehrotra
<ch...@gmail.com> wrote:
> That mode is fine for case like FileInputStream but not for case like
> S3 where underlying data is just a url and DataStore needs to make use
> of that.
Something like S3InputStream.getURL() should work just fine for that use case:
if (stream instanceof S3InputStream) {
URL url = ((S3InputStream) stream).getURL();
...; // create binary with S3 copy
}
The benefit of using InputStream instead of Binary for this case is
that the InputStream gets passed through the whole stack to the
underlying data store, whereas the Binary interface as a JCR construct
is interpreted higher up in oak-jcr.
BR,
Jukka Zitting
Re: Efficient import of binary data into Oak
Posted by Chetan Mehrotra <ch...@gmail.com>.
On Tue, Feb 18, 2014 at 2:32 PM, Jukka Zitting <ju...@gmail.com> wrote:
> Good point. That use case would probably be best handled with a
> specific InputStream subclass like suggested by Felix for files.
That mode is fine for case like FileInputStream but not for case like
S3 where underlying data is just a url and DataStore needs to make use
of that. Instead what I was referring to is the ability to pass custom
Binary implementations
- FileBinary implements javx.jcr.Binary -
- File object
- boolean flag indicating that ownership of file. If true it
indicates that client is transferring the ownership of the file to
Oak. In such a case the Oak DataStore can simply move the file
instance to its storage area
- S3Binary implements javx.jcr.Binary
- Signed S3 url which can be used by S3DataStore to perform Copy operation
This binary is then passed to JCR API and then DataStore makes use of
that and chooses an efficient path depending on the type of Binary
Couple of points to note
- Such an API would introduce a strong coupling with underlying DataStore
- But such an API would then be targeted for a very specific usecase
where the code needs to import high amount of binary data and it is
written keeping in mind the underlying DataStore being used such that
it can use the most optimized way
Chetan Mehrotra
Re: Efficient import of binary data into Oak
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Feb 18, 2014 at 2:25 AM, Chetan Mehrotra
<ch...@gmail.com> wrote:
> If we can have a way to create JCR Binary implementations which
> enables DataStore/BlobStore to efficiently transfer content then that
> would help.
ValueFactory.createBinary(InputStream stream)
The problem here, as far as I can see, is not within the JCR
implementation, but rather in the extra on-disk copy that FileUpload
does within Sling. I would suggest using the streaming API mentioned
by Ian.
> For example for File based DS the Binary created can keep a reference
> to the source File object and that Binary is used in JCR API.
That's already too late in the process. The temporary file should
never get created in the first place.
> Another example is S3DataStore - In some cases the file has already
> been transferred to S3 using other options. And the user wants to
> transfer the S3 file from its bucket to our bucket. So a Binary
> implementation which can just wrap the S3 url would enable the
> S3DataStore to transfer the content without streaming all content
> again [4]
Good point. That use case would probably be best handled with a
specific InputStream subclass like suggested by Felix for files.
> Any thoughts on the best way to enable users of Oak to create Binaries
> via other means (compared to current mode which only enables via
> InputStream) and enable the DataStores to make use of such binaries?
The functionality in JCR-3534 and OAK-834 is somewhat related to this.
BR,
Jukka Zitting
Re: Efficient import of binary data into Oak
Posted by Chetan Mehrotra <ch...@gmail.com>.
On Tue, Feb 18, 2014 at 2:36 PM, Ian Boston <ie...@tfd.co.uk> wrote:
> Upload the binary via some other mechanism then tell give Oak a
> pointer to the location, rather than giving Oak the binary.
Yup. As explained in mail before I am looking for "pass by reference"
semantics (somewhat related to JCR-3534).
Looking at OAK-834 further. I think we can use that. So instead of
custom Binary implementations (as suggested before) the required
information is "encoded" in the BinaryReference string and underlying
BlobStore can interpret such a url
For example to create a Binary out of File object and where client
code is transferring the ownership of file to Oak client code can
create a ReferenceBinary implementation with following string
file://host/path?own=true
The underlying BlobStore can
- Move the file to its working area
- OR efficient copy the file content
And then return the Blob
Similar scheme can be used by S3DataStore
http://<domainname>/objectname ?AWSAccessKeyId
=<accesskey>&Expires=<expire time>&Signature =<signature
string>&S3=true
Above approach can be used with current Oak implementations if
BlobStores can support such url.
Chetan Mehrotra
Re: Efficient import of binary data into Oak
Posted by Ian Boston <ie...@tfd.co.uk>.
Hi,
On 18 February 2014 08:50, Felix Meschberger <fm...@adobe.com> wrote:
> Hi
>
> That was my first thought, too: Nothing prevents the Binary implementation from checking whether the InputStream is a FileInputStream and then access the FileChannel from it.
>
> In the concrete case of Sling, the Sling RequestParameter.getInputStream() happens to call the Commons Upload FileItem.getInputStream() method which happens to return such a FileInputStream (if the item is actually stored in the filesystem, otherwise a ByteArrayInputStream happens to be returned).
I have vague memories of bypassing this behaviour and getting to the
stream before Sling touched it, but I could be mistaken.
Re-reading Chetans email at the end he asks:
"Any thoughts on the best way to enable users of Oak to create Binaries
via other means (compared to current mode which only enables via
InputStream) and enable the DataStores to make use of such binaries?"
Which sounds like:
Upload the binary via some other mechanism then tell give Oak a
pointer to the location, rather than giving Oak the binary.
Best Regards
Ian
>
> Regards
> Felix
>
> Am 18.02.2014 um 08:46 schrieb Ian Boston <ie...@tfd.co.uk>:
>
>> Hi,
>> Is there a reason you would not use the commons upload streaming Api to
>> connect the target output stream to the request stream? Iirc you can test
>> if both have nio channels and if the do just connect the two. I have used
>> this in the past to eliminate all GC activity and spooling. The streaming
>> Api is sensitive to order of the multiparts. You must use them as they
>> appear and not expect to be able to treat request parameters as a map. In
>> addition it is sensitive to other frameworks buffering or accessing the
>> request input stream.
>>
>> Best regards
>> Ian
>>
>> On Tuesday, February 18, 2014, Chetan Mehrotra <ch...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Currently in a Sling based application where a user uploads a file to
>>> the JCR following sequence of steps are executed
>>>
>>> 1. User uploads file via HTTP request mostly using Multi-Part form
>>> data based upload
>>>
>>> 2. Sling uses Commons File Upload to parse the multi-part request
>>> which uses a DiskFileItemFactory and write the binary content to a
>>> temporary file (for file size > 256 KB) [1]
>>>
>>> 3. Later the servlet would access the JCR Session and create a Binary
>>> value by extracting the InputStream
>>>
>>> 4. The file content would then be spooled into the BlobStore
>>>
>>> Effect of different blobstore
>>> ----------------------------------------
>>>
>>> Now depending on the type of BlobStore one of the following code flow
>>> would happen
>>>
>>> A - JR2 DataStores - The inputstream would be copied to file
>>> B - S3DataStore - The AWS SDK would be creating a temporary file and
>>> then that file content would be streamed back to the S3
>>> C - Segment - Content from InputStream would be stored as part of
>>> various segments
>>> D - MongoBlobStore - Content from InputStream would be pushed to
>>> remote mongo via multiple remote calls
>>>
>>> Things to note in above sequence
>>>
>>> 1. Uploaded content is copied twice.
>>> 2. The whole content is spooled via InputStream through JVM Heap
>>>
>>> Possible areas of Improvement
>>> --------------------------------
>>>
>>> 1. If the BlobStore is finally using some File (on same hard disk not
>>> NFS) then it might be better to *move* the file which was created in
>>> upload. This would help local FileDataStore and S3DataStore
>>>
>>> 2. Avoid spooling via InputStream if possible. Spooling via IS is slow
>>> [3]. Though in most cases we use efficient buffered copy which is
>>> marginally slower than NIO based variants. However avoiding moving
>>> byte[] might reduce pressure on GC (probably!)
>>>
>>> Changes required
>>> ------------------------
>>>
>>> If we can have a way to create JCR Binary implementations which
>>> enables DataStore/BlobStore to efficiently transfer content then that
>>> would help.
>>>
>>> For example for File based DS the Binary created can keep a reference
>>> to the source File object and that Binary is used in JCR API.
>>> Eventually the FileDataStore can treat it in a different way and move
>>> the file.
>>>
>>> Another example is S3DataStore - In some cases the file has already
>>> been transferred to S3 using other options. And the user wants to
>>> transfer the S3 file from its bucket to our bucket. So a Binary
>>> implementation which can just wrap the S3 url would enable the
>>> S3DataStore to transfer the content without streaming all content
>>> again [4]
>>>
>>> Any thoughts on the best way to enable users of Oak to create Binaries
>>> via other means (compared to current mode which only enables via
>>> InputStream) and enable the DataStores to make use of such binaries?
>>>
>>> Chetan Mehrotra
>>>
>>> [1]
>>> https://github.com/apache/sling/blob/trunk/bundles/engine/src/main/java/org/apache/sling/engine/impl/parameters/ParameterSupport.java#L190
>>> [2]
>>> http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/PutObjectRequest.html
>>> [3] http://www.baptiste-wicht.com/2010/08/file-copy-in-java-benchmark/3/
>>> [4]
>>> http://stackoverflow.com/questions/9664904/best-way-to-move-files-between-s3-buckets
>>>
>
Re: Efficient import of binary data into Oak
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Feb 18, 2014 at 3:50 AM, Felix Meschberger <fm...@adobe.com> wrote:
> That was my first thought, too: Nothing prevents the Binary implementation from checking
> whether the InputStream is a FileInputStream and then access the FileChannel from it.
Right, but then there's no easy way for the implementation to know
whether it's OK to just move the file away from the client as
suggested by Chetan. And while copying at the FileChannel level may be
a bit faster than via a normal InputStream, the benefit is minimal
compared to the overhead of spooling the upload to a temporary file.
As mentioned in the previous message, the best approach would be to
avoid the temporary file entirely.
BR,
Jukka Zitting
Re: Efficient import of binary data into Oak
Posted by Felix Meschberger <fm...@adobe.com>.
Hi
That was my first thought, too: Nothing prevents the Binary implementation from checking whether the InputStream is a FileInputStream and then access the FileChannel from it.
In the concrete case of Sling, the Sling RequestParameter.getInputStream() happens to call the Commons Upload FileItem.getInputStream() method which happens to return such a FileInputStream (if the item is actually stored in the filesystem, otherwise a ByteArrayInputStream happens to be returned).
Regards
Felix
Am 18.02.2014 um 08:46 schrieb Ian Boston <ie...@tfd.co.uk>:
> Hi,
> Is there a reason you would not use the commons upload streaming Api to
> connect the target output stream to the request stream? Iirc you can test
> if both have nio channels and if the do just connect the two. I have used
> this in the past to eliminate all GC activity and spooling. The streaming
> Api is sensitive to order of the multiparts. You must use them as they
> appear and not expect to be able to treat request parameters as a map. In
> addition it is sensitive to other frameworks buffering or accessing the
> request input stream.
>
> Best regards
> Ian
>
> On Tuesday, February 18, 2014, Chetan Mehrotra <ch...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Currently in a Sling based application where a user uploads a file to
>> the JCR following sequence of steps are executed
>>
>> 1. User uploads file via HTTP request mostly using Multi-Part form
>> data based upload
>>
>> 2. Sling uses Commons File Upload to parse the multi-part request
>> which uses a DiskFileItemFactory and write the binary content to a
>> temporary file (for file size > 256 KB) [1]
>>
>> 3. Later the servlet would access the JCR Session and create a Binary
>> value by extracting the InputStream
>>
>> 4. The file content would then be spooled into the BlobStore
>>
>> Effect of different blobstore
>> ----------------------------------------
>>
>> Now depending on the type of BlobStore one of the following code flow
>> would happen
>>
>> A - JR2 DataStores - The inputstream would be copied to file
>> B - S3DataStore - The AWS SDK would be creating a temporary file and
>> then that file content would be streamed back to the S3
>> C - Segment - Content from InputStream would be stored as part of
>> various segments
>> D - MongoBlobStore - Content from InputStream would be pushed to
>> remote mongo via multiple remote calls
>>
>> Things to note in above sequence
>>
>> 1. Uploaded content is copied twice.
>> 2. The whole content is spooled via InputStream through JVM Heap
>>
>> Possible areas of Improvement
>> --------------------------------
>>
>> 1. If the BlobStore is finally using some File (on same hard disk not
>> NFS) then it might be better to *move* the file which was created in
>> upload. This would help local FileDataStore and S3DataStore
>>
>> 2. Avoid spooling via InputStream if possible. Spooling via IS is slow
>> [3]. Though in most cases we use efficient buffered copy which is
>> marginally slower than NIO based variants. However avoiding moving
>> byte[] might reduce pressure on GC (probably!)
>>
>> Changes required
>> ------------------------
>>
>> If we can have a way to create JCR Binary implementations which
>> enables DataStore/BlobStore to efficiently transfer content then that
>> would help.
>>
>> For example for File based DS the Binary created can keep a reference
>> to the source File object and that Binary is used in JCR API.
>> Eventually the FileDataStore can treat it in a different way and move
>> the file.
>>
>> Another example is S3DataStore - In some cases the file has already
>> been transferred to S3 using other options. And the user wants to
>> transfer the S3 file from its bucket to our bucket. So a Binary
>> implementation which can just wrap the S3 url would enable the
>> S3DataStore to transfer the content without streaming all content
>> again [4]
>>
>> Any thoughts on the best way to enable users of Oak to create Binaries
>> via other means (compared to current mode which only enables via
>> InputStream) and enable the DataStores to make use of such binaries?
>>
>> Chetan Mehrotra
>>
>> [1]
>> https://github.com/apache/sling/blob/trunk/bundles/engine/src/main/java/org/apache/sling/engine/impl/parameters/ParameterSupport.java#L190
>> [2]
>> http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/PutObjectRequest.html
>> [3] http://www.baptiste-wicht.com/2010/08/file-copy-in-java-benchmark/3/
>> [4]
>> http://stackoverflow.com/questions/9664904/best-way-to-move-files-between-s3-buckets
>>
Re: Efficient import of binary data into Oak
Posted by Ian Boston <ie...@tfd.co.uk>.
Hi,
Is there a reason you would not use the commons upload streaming Api to
connect the target output stream to the request stream? Iirc you can test
if both have nio channels and if the do just connect the two. I have used
this in the past to eliminate all GC activity and spooling. The streaming
Api is sensitive to order of the multiparts. You must use them as they
appear and not expect to be able to treat request parameters as a map. In
addition it is sensitive to other frameworks buffering or accessing the
request input stream.
Best regards
Ian
On Tuesday, February 18, 2014, Chetan Mehrotra <ch...@gmail.com>
wrote:
> Hi,
>
> Currently in a Sling based application where a user uploads a file to
> the JCR following sequence of steps are executed
>
> 1. User uploads file via HTTP request mostly using Multi-Part form
> data based upload
>
> 2. Sling uses Commons File Upload to parse the multi-part request
> which uses a DiskFileItemFactory and write the binary content to a
> temporary file (for file size > 256 KB) [1]
>
> 3. Later the servlet would access the JCR Session and create a Binary
> value by extracting the InputStream
>
> 4. The file content would then be spooled into the BlobStore
>
> Effect of different blobstore
> ----------------------------------------
>
> Now depending on the type of BlobStore one of the following code flow
> would happen
>
> A - JR2 DataStores - The inputstream would be copied to file
> B - S3DataStore - The AWS SDK would be creating a temporary file and
> then that file content would be streamed back to the S3
> C - Segment - Content from InputStream would be stored as part of
> various segments
> D - MongoBlobStore - Content from InputStream would be pushed to
> remote mongo via multiple remote calls
>
> Things to note in above sequence
>
> 1. Uploaded content is copied twice.
> 2. The whole content is spooled via InputStream through JVM Heap
>
> Possible areas of Improvement
> --------------------------------
>
> 1. If the BlobStore is finally using some File (on same hard disk not
> NFS) then it might be better to *move* the file which was created in
> upload. This would help local FileDataStore and S3DataStore
>
> 2. Avoid spooling via InputStream if possible. Spooling via IS is slow
> [3]. Though in most cases we use efficient buffered copy which is
> marginally slower than NIO based variants. However avoiding moving
> byte[] might reduce pressure on GC (probably!)
>
> Changes required
> ------------------------
>
> If we can have a way to create JCR Binary implementations which
> enables DataStore/BlobStore to efficiently transfer content then that
> would help.
>
> For example for File based DS the Binary created can keep a reference
> to the source File object and that Binary is used in JCR API.
> Eventually the FileDataStore can treat it in a different way and move
> the file.
>
> Another example is S3DataStore - In some cases the file has already
> been transferred to S3 using other options. And the user wants to
> transfer the S3 file from its bucket to our bucket. So a Binary
> implementation which can just wrap the S3 url would enable the
> S3DataStore to transfer the content without streaming all content
> again [4]
>
> Any thoughts on the best way to enable users of Oak to create Binaries
> via other means (compared to current mode which only enables via
> InputStream) and enable the DataStores to make use of such binaries?
>
> Chetan Mehrotra
>
> [1]
> https://github.com/apache/sling/blob/trunk/bundles/engine/src/main/java/org/apache/sling/engine/impl/parameters/ParameterSupport.java#L190
> [2]
> http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/PutObjectRequest.html
> [3] http://www.baptiste-wicht.com/2010/08/file-copy-in-java-benchmark/3/
> [4]
> http://stackoverflow.com/questions/9664904/best-way-to-move-files-between-s3-buckets
>