You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@isis.apache.org by Kambiz Darabi <da...@m-creations.com> on 2016/08/03 16:37:28 UTC
Binary, non-base 64 upload
Hi,
I have to implement a legacy file upload service for a customer. Files
of up to some hundred megabytes are uploaded to a service over HTTP POST
and a unique ID is returned which can be used to refer to that file, add
metadata to it etc.
Using Blobs is not viable as parsing of the base64 encoded binary data
and the allocation of multiple strings during request processing
increases both the time and space requirements.
Ideally, I would be able to create an action on one of my domain
services which has a HttpServletRequest argument and take care of
reading the binary data and writing them to the file system.
I have played around with the request processing part of the
restfulobjects viewer/server and managed to get the request/response
objects through.
I know that this is a real edge case, but I think that it is important
to have solutions for edge cases, too, if Isis is to be used as a
general purpose platform.
Could some committers please review the changes and maybe propose a
cleaner way of implementing this?
https://github.com/m-creations/isis/commit/aa3b16a5cf463466f5abadbcc8cc73f16857a628
Thanks
Kambiz
Re: Binary, non-base 64 upload
Posted by Dan Haywood <da...@haywood-associates.co.uk>.
Thanks, Andy.
I've created ISIS-1479 [1] to capture this idea.
Dan
[1] https://issues.apache.org/jira/browse/ISIS-1479
On 26 August 2016 at 13:09, Andy Jefferson <an...@datanucleus.org> wrote:
> > With respect to the storage of the BLOBs themselves, I don't think (at
> > least I couldn't find) any datatype mapping to be able to map from a
> stream
> > (obtained from ServletInputStream) into a JDO entity, ie to avoid the
> > creation of the BLOB in-memory. If it *were* possible, then I don't
> think
> > Isis would care; just map the property as @Programmatic and Isis would
> > ignore it. Perhaps Andy might be able to chip in on this?
>
> java.io.File is currently the only java field type with a mapping that is
> streamable. See
> http://www.datanucleus.org/products/accessplatform_5_0/
> jdo/types.html#Other_Types
>
> Somebody could take existing code used for that type and contribute a
> mapping for some other java type.
> https://github.com/datanucleus/datanucleus-rdbms/
> blob/master/src/main/java/org/datanucleus/store/rdbms/mapping/datastore/
> BinaryStreamRDBMSMapping.java
>
>
>
> --
> Andy
> DataNucleus (Web: http://www.datanucleus.org Twitter: @datanucleus)
>
Re: Binary, non-base 64 upload
Posted by Andy Jefferson <an...@datanucleus.org>.
> With respect to the storage of the BLOBs themselves, I don't think (at
> least I couldn't find) any datatype mapping to be able to map from a stream
> (obtained from ServletInputStream) into a JDO entity, ie to avoid the
> creation of the BLOB in-memory. If it *were* possible, then I don't think
> Isis would care; just map the property as @Programmatic and Isis would
> ignore it. Perhaps Andy might be able to chip in on this?
java.io.File is currently the only java field type with a mapping that is streamable. See
http://www.datanucleus.org/products/accessplatform_5_0/jdo/types.html#Other_Types
Somebody could take existing code used for that type and contribute a mapping for some other java type.
https://github.com/datanucleus/datanucleus-rdbms/blob/master/src/main/java/org/datanucleus/store/rdbms/mapping/datastore/BinaryStreamRDBMSMapping.java
--
Andy
DataNucleus (Web: http://www.datanucleus.org Twitter: @datanucleus)
Re: Binary, non-base 64 upload
Posted by Dan Haywood <da...@haywood-associates.co.uk>.
Coming late to this thread, apologies (been on vacation).
With respect to a mechanism for uploading BLOBs via RO, in the first event
I think I'd rather this is done through separate server-side resources that
sit alongside the Isis runtime. Using the headless APIs [1] it ought to be
relatively easy to have a custom controller get access into the Isis
runtime. (I'm hoping to make the handling of sessions and transactions
more consistent between RO and Wicket viewer, eg through a standard set of
filters in web.xml; hopefully that will make it easier for us to support
custom functionality, eg this and no-wicket).
With respect to the storage of the BLOBs themselves, I don't think (at
least I couldn't find) any datatype mapping to be able to map from a stream
(obtained from ServletInputStream) into a JDO entity, ie to avoid the
creation of the BLOB in-memory. If it *were* possible, then I don't think
Isis would care; just map the property as @Programmatic and Isis would
ignore it. Perhaps Andy might be able to chip in on this?
Somewhat related, within Estatio we have a requirement to generate PDFs
(invoices) but want to eventually store these outside of the RDBMS, eg S3
or in an external CMS. Since our BLOBs aren't too large, we are
considering defining an entity that stores the BLOBs locally, but then have
a Camel or Quartz background job that would then move the BLOB data out
into the external storage. When the BLOB data is moved, it would then hold
a reference to the S3 BLOB. The user experience is that the PDF would be
immediately available (stored within the same transaction that creates the
invoice), and without all the hassle of having to handle eventual
consistency issues.
HTH
Dan
[1] http://isis.apache.org/guides/ugbtb.html#_ugbtb_headless-access
[2] https://github.com/subes/invesdwin-nowicket
On 6 August 2016 at 09:04, Kambiz Darabi <da...@m-creations.com> wrote:
> Hi David,
>
> my solution ideally needs to support these storage backends
>
> - file system
> - Swift
> - SoftLayer Object Storage
>
> So, once I have a way of shoe-horning a raw binary POST into Isis, which
> I wouldn't have to read fully into memory, I would code the rest around
> JClouds [3] with its BlobStore [4], which supports (and glosses over the
> differences between):
>
> - AWS S3
> - Google Cloud Storage
> - Microsoft Azure
> - Rackspace US
> - Rackspace UK
>
> The rest of it could be coded as a blob-storage module with one or two
> tables which store storage paths, blob IDs, etc. like the security
> module does for users/groups.
>
> Cheers
>
>
> Kambiz
>
> [1] http://docs.openstack.org/developer/swift/
> [2] http://www.softlayer.com/object-storage
> [3] https://jclouds.apache.org/
> [4] https://jclouds.apache.org/start/blobstore/
>
> On 2016-08-06 08:23 CEST, David Tildesley <da...@yahoo.co.nz> wrote:
>
> > Good point Kambiz - with the likes of S3 object storage this is
> > becoming a common requirement - especially for very large files where
> > loading the whole file into memory is not appropriate. And much better
> > than storing in a rdbms blob column.
> >
> > I suggest a hidden mandatory attribute on a domain object (actually to
> > hold the stored object ID once it has uploaded) to prevent the form
> > from being submitted and javascript (delivered via a custom wicket
> > component) using Fine Uploader Javascript Upload Library since it is
> > Ajax your form is not submitted until your javascript allows it by
> > storing the uploaded object ID in the hidden mandatory field. Also the
> > user sees a fine progress bar whilst uploading and this library
> > supports chunking and direct to S3 and Azure or to traditional server
> > side (servlet).
> >
> > What do you think?
> >
> > Regards,
> > David.
> >
> >
> > image
> > Fine Uploader Javascript Upload Library
> > Fine Uploader. A dependency-free, open-source, native browser
> > upload tool.
> >
> > View on fineuploader.com
> >
> >
> > image
> >
> > On Saturday, 6 August 2016 7:31 AM, Martin Grigorov
> > <mg...@apache.org> wrote:
> >
> > Hi Kambiz,
> >
> > I guess this code works only because you don't need another request
> > parameter in the same action.
> > Latest versions of Isis recommend to have an action per request
> > parameter,
> > so maybe this is not an issue.
> > But imagine submitting a form with one or more text fields and one
> > file
> > upload field. To be able to pass the values of the input fields Isis
> > will
> > need to call request.getParameter(someName) and this will consume the
> > request body and later your code won't see anything in the
> > inputstream.
> >
> > Martin Grigorov
> > Wicket Training and Consulting
> > https://twitter.com/mtgrigorov
> >
> > On Fri, Aug 5, 2016 at 4:27 PM, Kambiz Darabi <da...@m-creations.com>
> > wrote:
> >
> >> Hello Martin,
> >>
> >> On 2016-08-05 13:42 CEST, Martin Grigorov <mg...@apache.org>
> > wrote:
> >>
> >> > [...]
> >> > The problem here is that you have to make sure that your code is
> > the very
> >> > first one that reads from the ServletInputStream. Otherwise the
> > body will
> >> > be already consumed.
> >> > You may need to override some Wicket/RESTeasy code to be able to
> > do this
> >> > for the default viewers.
> >> > And this would be much harder than custom Servlet Filter in front
> > of the
> >> > ones by Wicket/RESTeasy
> >>
> >> If you look at the change in my original post:
> >>
> >>
> > https://github.com/m-creations/isis/commit/
> aa3b16a5cf463466f5abadbcc8cc73
> >> f16857a628
> >>
> >> you can see that I added this functionality to the
> >> restfulobjects-viewer, so that I can be sure that the full stream is
> >> handed to the action without any change.
> >>
> >> That code is tested against the Isis archetype with this addition to
> > the
> >> SimpleObjectMenu service:
> >>
> >> @Action(semantics = SemanticsOf.SAFE)
> >> public void upload(HttpServletRequest request, HttpServletResponse
> >> response) {
> >> FileOutputStream out = null;
> >> try {
> >> File outFile = new File("/tmp/request");
> >> out = new FileOutputStream(outFile, false);
> >> IOUtils.copy(request.getInputStream(), out);
> >> out.close();
> >> } catch (IOException e) {
> >> if(out != null) {
> >> try {
> >> out.close();
> >> } catch (IOException e1) {
> >> // LOG something
> >> }
> >> }
> >> // throw something
> >> }
> >> }
> >>
> >>
> >> and the file in /tmp/request is identical to the posted binary file
> > which
> >> is sent by
> >> curl:
> >>
> >> curl -X POST --data-binary @big-binary-file
> >> --header "Authorization: Basic c3ZlbjpwYXNz" \
> >> --header "Accept: application/json;profile=urn:org.apache.isis/v1" \
> >> --header "Content-Type: application/octet-stream" \
> >> http://localhost:8080/restful/services/SimpleObjectMenu/
> >> actions/upload/invoke
> >>
> >>
> >> Cheers
> >>
> >>
> >> Kambiz
> >>
> >> >
> >> >> action with the metadata of the files/request, but it would help
> > a lot
> >> >> to have such a facility inside Isis instead of writing a separate
> >> >> servlet and manually integrating its deployment and the calling
> > of
> >> >> domain actions etc.
> >> >>
> >> >> Cheers
> >> >>
> >> >>
> >> >> Kambiz
> >> >>
> >> >>
> >> >> On 2016-08-03 18:53 CEST, Willie Loyd Tandingan <
> >> tandingan.wlb@gmail.com>
> >> >> wrote:
> >> >>
> >> >> > Hi Kambiz,
> >> >> >
> >> >> > We had a requirement similar to yours. The problem with
> > injecting
> >> >> > HttpServletRequest to domain services is that it kind of
> > violates the
> >> >> DDD's
> >> >> > hexagonal architecture by letting viewer implementation leak
> > into the
> >> >> > domain layer.
> >> >> >
> >> >> > The specific requirement that we had was to be able to
> > pre-upload
> >> files
> >> >> > before submitting an action. We developed a custom blob that
> > stores
> >> an id
> >> >> > instead of the actual data, a service layer that manages a blob
> >> storage
> >> >> > abstracting and running on top of jclouds (we plan to be able
> > to
> >> support
> >> >> > other types of storage aside from file system), and a file
> > service
> >> that
> >> >> > manages the state of "transient files" (files that were
> > pre-uploaded
> >> or
> >> >> > that are returned from actions as result of export functions,
> > etc.) as
> >> >> well
> >> >> > as means to persist them (i.e. move them to another
> >> >> > container/bucket/folder).
> >> >> >
> >> >> > We then developed a custom servlet that handles download and
> > upload
> >> and
> >> >> > integrate them with isis. This enabled us to implement features
> > like
> >> >> forced
> >> >> > attachment content disposition, or on-demand image resizing.
> >> >> >
> >> >> > Should you have other ideas or questions, feel free to ask.
> >> >> >
> >> >> >
> >> >> > Best regards,
> >> >> > Willie
> >> >> > On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi
> > <da...@m-creations.com>
> >> >> wrote:
> >> >> >
> >> >> >> Hi,
> >> >> >>
> >> >> >> I have to implement a legacy file upload service for a
> > customer.
> >> Files
> >> >> >> of up to some hundred megabytes are uploaded to a service over
> > HTTP
> >> POST
> >> >> >> and a unique ID is returned which can be used to refer to that
> > file,
> >> add
> >> >> >> metadata to it etc.
> >> >> >>
> >> >> >> Using Blobs is not viable as parsing of the base64 encoded
> > binary
> >> data
> >> >> >> and the allocation of multiple strings during request
> > processing
> >> >> >> increases both the time and space requirements.
> >> >> >>
> >> >> >> Ideally, I would be able to create an action on one of my
> > domain
> >> >> >> services which has a HttpServletRequest argument and take care
> > of
> >> >> >> reading the binary data and writing them to the file system.
> >> >> >>
> >> >> >> I have played around with the request processing part of the
> >> >> >> restfulobjects viewer/server and managed to get the
> > request/response
> >> >> >> objects through.
> >> >> >>
> >> >> >> I know that this is a real edge case, but I think that it is
> >> important
> >> >> >> to have solutions for edge cases, too, if Isis is to be used
> > as a
> >> >> >> general purpose platform.
> >> >> >>
> >> >> >> Could some committers please review the changes and maybe
> > propose a
> >> >> >> cleaner way of implementing this?
> >> >> >>
> >> >> >>
> >> >> >> https://github.com/m-creations/isis/commit/
> >> >> aa3b16a5cf463466f5abadbcc8cc73f16857a628
> >> >> >>
> >> >> >> Thanks
> >> >> >>
> >> >> >>
> >> >> >> Kambiz
> >> >> >>
> >> >> >>
> >> >>
> >>
>
Re: Binary, non-base 64 upload
Posted by Kambiz Darabi <da...@m-creations.com>.
Hi David,
my solution ideally needs to support these storage backends
- file system
- Swift
- SoftLayer Object Storage
So, once I have a way of shoe-horning a raw binary POST into Isis, which
I wouldn't have to read fully into memory, I would code the rest around
JClouds [3] with its BlobStore [4], which supports (and glosses over the
differences between):
- AWS S3
- Google Cloud Storage
- Microsoft Azure
- Rackspace US
- Rackspace UK
The rest of it could be coded as a blob-storage module with one or two
tables which store storage paths, blob IDs, etc. like the security
module does for users/groups.
Cheers
Kambiz
[1] http://docs.openstack.org/developer/swift/
[2] http://www.softlayer.com/object-storage
[3] https://jclouds.apache.org/
[4] https://jclouds.apache.org/start/blobstore/
On 2016-08-06 08:23 CEST, David Tildesley <da...@yahoo.co.nz> wrote:
> Good point Kambiz - with the likes of S3 object storage this is
> becoming a common requirement - especially for very large files where
> loading the whole file into memory is not appropriate. And much better
> than storing in a rdbms blob column.
>
> I suggest a hidden mandatory attribute on a domain object (actually to
> hold the stored object ID once it has uploaded) to prevent the form
> from being submitted and javascript (delivered via a custom wicket
> component) using Fine Uploader Javascript Upload Library since it is
> Ajax your form is not submitted until your javascript allows it by
> storing the uploaded object ID in the hidden mandatory field. Also the
> user sees a fine progress bar whilst uploading and this library
> supports chunking and direct to S3 and Azure or to traditional server
> side (servlet).
>
> What do you think?
>
> Regards,
> David.
>
>
> image
> Fine Uploader Javascript Upload Library
> Fine Uploader. A dependency-free, open-source, native browser
> upload tool.
>
> View on fineuploader.com
>
>
> image
>
> On Saturday, 6 August 2016 7:31 AM, Martin Grigorov
> <mg...@apache.org> wrote:
>
> Hi Kambiz,
>
> I guess this code works only because you don't need another request
> parameter in the same action.
> Latest versions of Isis recommend to have an action per request
> parameter,
> so maybe this is not an issue.
> But imagine submitting a form with one or more text fields and one
> file
> upload field. To be able to pass the values of the input fields Isis
> will
> need to call request.getParameter(someName) and this will consume the
> request body and later your code won't see anything in the
> inputstream.
>
> Martin Grigorov
> Wicket Training and Consulting
> https://twitter.com/mtgrigorov
>
> On Fri, Aug 5, 2016 at 4:27 PM, Kambiz Darabi <da...@m-creations.com>
> wrote:
>
>> Hello Martin,
>>
>> On 2016-08-05 13:42 CEST, Martin Grigorov <mg...@apache.org>
> wrote:
>>
>> > [...]
>> > The problem here is that you have to make sure that your code is
> the very
>> > first one that reads from the ServletInputStream. Otherwise the
> body will
>> > be already consumed.
>> > You may need to override some Wicket/RESTeasy code to be able to
> do this
>> > for the default viewers.
>> > And this would be much harder than custom Servlet Filter in front
> of the
>> > ones by Wicket/RESTeasy
>>
>> If you look at the change in my original post:
>>
>>
> https://github.com/m-creations/isis/commit/aa3b16a5cf463466f5abadbcc8cc73
>> f16857a628
>>
>> you can see that I added this functionality to the
>> restfulobjects-viewer, so that I can be sure that the full stream is
>> handed to the action without any change.
>>
>> That code is tested against the Isis archetype with this addition to
> the
>> SimpleObjectMenu service:
>>
>> @Action(semantics = SemanticsOf.SAFE)
>> public void upload(HttpServletRequest request, HttpServletResponse
>> response) {
>> FileOutputStream out = null;
>> try {
>> File outFile = new File("/tmp/request");
>> out = new FileOutputStream(outFile, false);
>> IOUtils.copy(request.getInputStream(), out);
>> out.close();
>> } catch (IOException e) {
>> if(out != null) {
>> try {
>> out.close();
>> } catch (IOException e1) {
>> // LOG something
>> }
>> }
>> // throw something
>> }
>> }
>>
>>
>> and the file in /tmp/request is identical to the posted binary file
> which
>> is sent by
>> curl:
>>
>> curl -X POST --data-binary @big-binary-file
>> --header "Authorization: Basic c3ZlbjpwYXNz" \
>> --header "Accept: application/json;profile=urn:org.apache.isis/v1" \
>> --header "Content-Type: application/octet-stream" \
>> http://localhost:8080/restful/services/SimpleObjectMenu/
>> actions/upload/invoke
>>
>>
>> Cheers
>>
>>
>> Kambiz
>>
>> >
>> >> action with the metadata of the files/request, but it would help
> a lot
>> >> to have such a facility inside Isis instead of writing a separate
>> >> servlet and manually integrating its deployment and the calling
> of
>> >> domain actions etc.
>> >>
>> >> Cheers
>> >>
>> >>
>> >> Kambiz
>> >>
>> >>
>> >> On 2016-08-03 18:53 CEST, Willie Loyd Tandingan <
>> tandingan.wlb@gmail.com>
>> >> wrote:
>> >>
>> >> > Hi Kambiz,
>> >> >
>> >> > We had a requirement similar to yours. The problem with
> injecting
>> >> > HttpServletRequest to domain services is that it kind of
> violates the
>> >> DDD's
>> >> > hexagonal architecture by letting viewer implementation leak
> into the
>> >> > domain layer.
>> >> >
>> >> > The specific requirement that we had was to be able to
> pre-upload
>> files
>> >> > before submitting an action. We developed a custom blob that
> stores
>> an id
>> >> > instead of the actual data, a service layer that manages a blob
>> storage
>> >> > abstracting and running on top of jclouds (we plan to be able
> to
>> support
>> >> > other types of storage aside from file system), and a file
> service
>> that
>> >> > manages the state of "transient files" (files that were
> pre-uploaded
>> or
>> >> > that are returned from actions as result of export functions,
> etc.) as
>> >> well
>> >> > as means to persist them (i.e. move them to another
>> >> > container/bucket/folder).
>> >> >
>> >> > We then developed a custom servlet that handles download and
> upload
>> and
>> >> > integrate them with isis. This enabled us to implement features
> like
>> >> forced
>> >> > attachment content disposition, or on-demand image resizing.
>> >> >
>> >> > Should you have other ideas or questions, feel free to ask.
>> >> >
>> >> >
>> >> > Best regards,
>> >> > Willie
>> >> > On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi
> <da...@m-creations.com>
>> >> wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> I have to implement a legacy file upload service for a
> customer.
>> Files
>> >> >> of up to some hundred megabytes are uploaded to a service over
> HTTP
>> POST
>> >> >> and a unique ID is returned which can be used to refer to that
> file,
>> add
>> >> >> metadata to it etc.
>> >> >>
>> >> >> Using Blobs is not viable as parsing of the base64 encoded
> binary
>> data
>> >> >> and the allocation of multiple strings during request
> processing
>> >> >> increases both the time and space requirements.
>> >> >>
>> >> >> Ideally, I would be able to create an action on one of my
> domain
>> >> >> services which has a HttpServletRequest argument and take care
> of
>> >> >> reading the binary data and writing them to the file system.
>> >> >>
>> >> >> I have played around with the request processing part of the
>> >> >> restfulobjects viewer/server and managed to get the
> request/response
>> >> >> objects through.
>> >> >>
>> >> >> I know that this is a real edge case, but I think that it is
>> important
>> >> >> to have solutions for edge cases, too, if Isis is to be used
> as a
>> >> >> general purpose platform.
>> >> >>
>> >> >> Could some committers please review the changes and maybe
> propose a
>> >> >> cleaner way of implementing this?
>> >> >>
>> >> >>
>> >> >> https://github.com/m-creations/isis/commit/
>> >> aa3b16a5cf463466f5abadbcc8cc73f16857a628
>> >> >>
>> >> >> Thanks
>> >> >>
>> >> >>
>> >> >> Kambiz
>> >> >>
>> >> >>
>> >>
>>
Re: Binary, non-base 64 upload
Posted by David Tildesley <da...@yahoo.co.nz.INVALID>.
Good point Kambiz - with the likes of S3 object storage this is becoming a common requirement - especially for very large files where loading the whole file into memory is not appropriate. And much better than storing in a rdbms blob column.
I suggest a hidden mandatory attribute on a domain object (actually to hold the stored object ID once it has uploaded) to prevent the form from being submitted and javascript (delivered via a custom wicket component) using Fine Uploader Javascript Upload Library since it is Ajax your form is not submitted until your javascript allows it by storing the uploaded object ID in the hidden mandatory field. Also the user sees a fine progress bar whilst uploading and this library supports chunking and direct to S3 and Azure or to traditional server side (servlet).
What do you think?
Regards,David.
| |
| | | | | | | |
| Fine Uploader Javascript Upload LibraryFine Uploader. A dependency-free, open-source, native browser upload tool. |
| |
| View on fineuploader.com | Preview by Yahoo |
| |
| |
On Saturday, 6 August 2016 7:31 AM, Martin Grigorov <mg...@apache.org> wrote:
Hi Kambiz,
I guess this code works only because you don't need another request
parameter in the same action.
Latest versions of Isis recommend to have an action per request parameter,
so maybe this is not an issue.
But imagine submitting a form with one or more text fields and one file
upload field. To be able to pass the values of the input fields Isis will
need to call request.getParameter(someName) and this will consume the
request body and later your code won't see anything in the inputstream.
Martin Grigorov
Wicket Training and Consulting
https://twitter.com/mtgrigorov
On Fri, Aug 5, 2016 at 4:27 PM, Kambiz Darabi <da...@m-creations.com>
wrote:
> Hello Martin,
>
> On 2016-08-05 13:42 CEST, Martin Grigorov <mg...@apache.org> wrote:
>
> > [...]
> > The problem here is that you have to make sure that your code is the very
> > first one that reads from the ServletInputStream. Otherwise the body will
> > be already consumed.
> > You may need to override some Wicket/RESTeasy code to be able to do this
> > for the default viewers.
> > And this would be much harder than custom Servlet Filter in front of the
> > ones by Wicket/RESTeasy
>
> If you look at the change in my original post:
>
> https://github.com/m-creations/isis/commit/aa3b16a5cf463466f5abadbcc8cc73
> f16857a628
>
> you can see that I added this functionality to the
> restfulobjects-viewer, so that I can be sure that the full stream is
> handed to the action without any change.
>
> That code is tested against the Isis archetype with this addition to the
> SimpleObjectMenu service:
>
> @Action(semantics = SemanticsOf.SAFE)
> public void upload(HttpServletRequest request, HttpServletResponse
> response) {
> FileOutputStream out = null;
> try {
> File outFile = new File("/tmp/request");
> out = new FileOutputStream(outFile, false);
> IOUtils.copy(request.getInputStream(), out);
> out.close();
> } catch (IOException e) {
> if(out != null) {
> try {
> out.close();
> } catch (IOException e1) {
> // LOG something
> }
> }
> // throw something
> }
> }
>
>
> and the file in /tmp/request is identical to the posted binary file which
> is sent by
> curl:
>
> curl -X POST --data-binary @big-binary-file
> --header "Authorization: Basic c3ZlbjpwYXNz" \
> --header "Accept: application/json;profile=urn:org.apache.isis/v1" \
> --header "Content-Type: application/octet-stream" \
> http://localhost:8080/restful/services/SimpleObjectMenu/
> actions/upload/invoke
>
>
> Cheers
>
>
> Kambiz
>
> >
> >> action with the metadata of the files/request, but it would help a lot
> >> to have such a facility inside Isis instead of writing a separate
> >> servlet and manually integrating its deployment and the calling of
> >> domain actions etc.
> >>
> >> Cheers
> >>
> >>
> >> Kambiz
> >>
> >>
> >> On 2016-08-03 18:53 CEST, Willie Loyd Tandingan <
> tandingan.wlb@gmail.com>
> >> wrote:
> >>
> >> > Hi Kambiz,
> >> >
> >> > We had a requirement similar to yours. The problem with injecting
> >> > HttpServletRequest to domain services is that it kind of violates the
> >> DDD's
> >> > hexagonal architecture by letting viewer implementation leak into the
> >> > domain layer.
> >> >
> >> > The specific requirement that we had was to be able to pre-upload
> files
> >> > before submitting an action. We developed a custom blob that stores
> an id
> >> > instead of the actual data, a service layer that manages a blob
> storage
> >> > abstracting and running on top of jclouds (we plan to be able to
> support
> >> > other types of storage aside from file system), and a file service
> that
> >> > manages the state of "transient files" (files that were pre-uploaded
> or
> >> > that are returned from actions as result of export functions, etc.) as
> >> well
> >> > as means to persist them (i.e. move them to another
> >> > container/bucket/folder).
> >> >
> >> > We then developed a custom servlet that handles download and upload
> and
> >> > integrate them with isis. This enabled us to implement features like
> >> forced
> >> > attachment content disposition, or on-demand image resizing.
> >> >
> >> > Should you have other ideas or questions, feel free to ask.
> >> >
> >> >
> >> > Best regards,
> >> > Willie
> >> > On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi <da...@m-creations.com>
> >> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I have to implement a legacy file upload service for a customer.
> Files
> >> >> of up to some hundred megabytes are uploaded to a service over HTTP
> POST
> >> >> and a unique ID is returned which can be used to refer to that file,
> add
> >> >> metadata to it etc.
> >> >>
> >> >> Using Blobs is not viable as parsing of the base64 encoded binary
> data
> >> >> and the allocation of multiple strings during request processing
> >> >> increases both the time and space requirements.
> >> >>
> >> >> Ideally, I would be able to create an action on one of my domain
> >> >> services which has a HttpServletRequest argument and take care of
> >> >> reading the binary data and writing them to the file system.
> >> >>
> >> >> I have played around with the request processing part of the
> >> >> restfulobjects viewer/server and managed to get the request/response
> >> >> objects through.
> >> >>
> >> >> I know that this is a real edge case, but I think that it is
> important
> >> >> to have solutions for edge cases, too, if Isis is to be used as a
> >> >> general purpose platform.
> >> >>
> >> >> Could some committers please review the changes and maybe propose a
> >> >> cleaner way of implementing this?
> >> >>
> >> >>
> >> >> https://github.com/m-creations/isis/commit/
> >> aa3b16a5cf463466f5abadbcc8cc73f16857a628
> >> >>
> >> >> Thanks
> >> >>
> >> >>
> >> >> Kambiz
> >> >>
> >> >>
> >>
>
Re: Binary, non-base 64 upload
Posted by Kambiz Darabi <da...@m-creations.com>.
Hi Martin,
sorry for not stating it explicitly: this functionality is only for
domain services with VIEW_REST_ONLY nature and it doesn't make sense to
mix HttpServletRequest with any other action parameter. The full
implementation would have to verify this during annotation processing.
It was probably distracting to put the method into SimpleObjectMenu, it
belongs in a new domain service which is only visible in the restful
objects viewer.
Sorry for the confusion.
Kambiz
On 2016-08-05 21:30 CEST, Martin Grigorov <mg...@apache.org> wrote:
> Hi Kambiz,
>
> I guess this code works only because you don't need another request
> parameter in the same action.
> Latest versions of Isis recommend to have an action per request parameter,
> so maybe this is not an issue.
> But imagine submitting a form with one or more text fields and one file
> upload field. To be able to pass the values of the input fields Isis will
> need to call request.getParameter(someName) and this will consume the
> request body and later your code won't see anything in the inputstream.
>
> Martin Grigorov
> Wicket Training and Consulting
> https://twitter.com/mtgrigorov
>
> On Fri, Aug 5, 2016 at 4:27 PM, Kambiz Darabi <da...@m-creations.com>
> wrote:
>
>> Hello Martin,
>>
>> On 2016-08-05 13:42 CEST, Martin Grigorov <mg...@apache.org> wrote:
>>
>> > [...]
>> > The problem here is that you have to make sure that your code is the very
>> > first one that reads from the ServletInputStream. Otherwise the body will
>> > be already consumed.
>> > You may need to override some Wicket/RESTeasy code to be able to do this
>> > for the default viewers.
>> > And this would be much harder than custom Servlet Filter in front of the
>> > ones by Wicket/RESTeasy
>>
>> If you look at the change in my original post:
>>
>> https://github.com/m-creations/isis/commit/aa3b16a5cf463466f5abadbcc8cc73
>> f16857a628
>>
>> you can see that I added this functionality to the
>> restfulobjects-viewer, so that I can be sure that the full stream is
>> handed to the action without any change.
>>
>> That code is tested against the Isis archetype with this addition to the
>> SimpleObjectMenu service:
>>
>> @Action(semantics = SemanticsOf.SAFE)
>> public void upload(HttpServletRequest request, HttpServletResponse
>> response) {
>> FileOutputStream out = null;
>> try {
>> File outFile = new File("/tmp/request");
>> out = new FileOutputStream(outFile, false);
>> IOUtils.copy(request.getInputStream(), out);
>> out.close();
>> } catch (IOException e) {
>> if(out != null) {
>> try {
>> out.close();
>> } catch (IOException e1) {
>> // LOG something
>> }
>> }
>> // throw something
>> }
>> }
>>
>>
>> and the file in /tmp/request is identical to the posted binary file which
>> is sent by
>> curl:
>>
>> curl -X POST --data-binary @big-binary-file
>> --header "Authorization: Basic c3ZlbjpwYXNz" \
>> --header "Accept: application/json;profile=urn:org.apache.isis/v1" \
>> --header "Content-Type: application/octet-stream" \
>> http://localhost:8080/restful/services/SimpleObjectMenu/
>> actions/upload/invoke
>>
>>
>> Cheers
>>
>>
>> Kambiz
>>
>> >
>> >> action with the metadata of the files/request, but it would help a lot
>> >> to have such a facility inside Isis instead of writing a separate
>> >> servlet and manually integrating its deployment and the calling of
>> >> domain actions etc.
>> >>
>> >> Cheers
>> >>
>> >>
>> >> Kambiz
>> >>
>> >>
>> >> On 2016-08-03 18:53 CEST, Willie Loyd Tandingan <
>> tandingan.wlb@gmail.com>
>> >> wrote:
>> >>
>> >> > Hi Kambiz,
>> >> >
>> >> > We had a requirement similar to yours. The problem with injecting
>> >> > HttpServletRequest to domain services is that it kind of violates the
>> >> DDD's
>> >> > hexagonal architecture by letting viewer implementation leak into the
>> >> > domain layer.
>> >> >
>> >> > The specific requirement that we had was to be able to pre-upload
>> files
>> >> > before submitting an action. We developed a custom blob that stores
>> an id
>> >> > instead of the actual data, a service layer that manages a blob
>> storage
>> >> > abstracting and running on top of jclouds (we plan to be able to
>> support
>> >> > other types of storage aside from file system), and a file service
>> that
>> >> > manages the state of "transient files" (files that were pre-uploaded
>> or
>> >> > that are returned from actions as result of export functions, etc.) as
>> >> well
>> >> > as means to persist them (i.e. move them to another
>> >> > container/bucket/folder).
>> >> >
>> >> > We then developed a custom servlet that handles download and upload
>> and
>> >> > integrate them with isis. This enabled us to implement features like
>> >> forced
>> >> > attachment content disposition, or on-demand image resizing.
>> >> >
>> >> > Should you have other ideas or questions, feel free to ask.
>> >> >
>> >> >
>> >> > Best regards,
>> >> > Willie
>> >> > On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi <da...@m-creations.com>
>> >> wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> I have to implement a legacy file upload service for a customer.
>> Files
>> >> >> of up to some hundred megabytes are uploaded to a service over HTTP
>> POST
>> >> >> and a unique ID is returned which can be used to refer to that file,
>> add
>> >> >> metadata to it etc.
>> >> >>
>> >> >> Using Blobs is not viable as parsing of the base64 encoded binary
>> data
>> >> >> and the allocation of multiple strings during request processing
>> >> >> increases both the time and space requirements.
>> >> >>
>> >> >> Ideally, I would be able to create an action on one of my domain
>> >> >> services which has a HttpServletRequest argument and take care of
>> >> >> reading the binary data and writing them to the file system.
>> >> >>
>> >> >> I have played around with the request processing part of the
>> >> >> restfulobjects viewer/server and managed to get the request/response
>> >> >> objects through.
>> >> >>
>> >> >> I know that this is a real edge case, but I think that it is
>> important
>> >> >> to have solutions for edge cases, too, if Isis is to be used as a
>> >> >> general purpose platform.
>> >> >>
>> >> >> Could some committers please review the changes and maybe propose a
>> >> >> cleaner way of implementing this?
>> >> >>
>> >> >>
>> >> >> https://github.com/m-creations/isis/commit/
>> >> aa3b16a5cf463466f5abadbcc8cc73f16857a628
>> >> >>
>> >> >> Thanks
>> >> >>
>> >> >>
>> >> >> Kambiz
>> >> >>
>> >> >>
>> >>
>>
Re: Binary, non-base 64 upload
Posted by Martin Grigorov <mg...@apache.org>.
Hi Kambiz,
I guess this code works only because you don't need another request
parameter in the same action.
Latest versions of Isis recommend to have an action per request parameter,
so maybe this is not an issue.
But imagine submitting a form with one or more text fields and one file
upload field. To be able to pass the values of the input fields Isis will
need to call request.getParameter(someName) and this will consume the
request body and later your code won't see anything in the inputstream.
Martin Grigorov
Wicket Training and Consulting
https://twitter.com/mtgrigorov
On Fri, Aug 5, 2016 at 4:27 PM, Kambiz Darabi <da...@m-creations.com>
wrote:
> Hello Martin,
>
> On 2016-08-05 13:42 CEST, Martin Grigorov <mg...@apache.org> wrote:
>
> > [...]
> > The problem here is that you have to make sure that your code is the very
> > first one that reads from the ServletInputStream. Otherwise the body will
> > be already consumed.
> > You may need to override some Wicket/RESTeasy code to be able to do this
> > for the default viewers.
> > And this would be much harder than custom Servlet Filter in front of the
> > ones by Wicket/RESTeasy
>
> If you look at the change in my original post:
>
> https://github.com/m-creations/isis/commit/aa3b16a5cf463466f5abadbcc8cc73
> f16857a628
>
> you can see that I added this functionality to the
> restfulobjects-viewer, so that I can be sure that the full stream is
> handed to the action without any change.
>
> That code is tested against the Isis archetype with this addition to the
> SimpleObjectMenu service:
>
> @Action(semantics = SemanticsOf.SAFE)
> public void upload(HttpServletRequest request, HttpServletResponse
> response) {
> FileOutputStream out = null;
> try {
> File outFile = new File("/tmp/request");
> out = new FileOutputStream(outFile, false);
> IOUtils.copy(request.getInputStream(), out);
> out.close();
> } catch (IOException e) {
> if(out != null) {
> try {
> out.close();
> } catch (IOException e1) {
> // LOG something
> }
> }
> // throw something
> }
> }
>
>
> and the file in /tmp/request is identical to the posted binary file which
> is sent by
> curl:
>
> curl -X POST --data-binary @big-binary-file
> --header "Authorization: Basic c3ZlbjpwYXNz" \
> --header "Accept: application/json;profile=urn:org.apache.isis/v1" \
> --header "Content-Type: application/octet-stream" \
> http://localhost:8080/restful/services/SimpleObjectMenu/
> actions/upload/invoke
>
>
> Cheers
>
>
> Kambiz
>
> >
> >> action with the metadata of the files/request, but it would help a lot
> >> to have such a facility inside Isis instead of writing a separate
> >> servlet and manually integrating its deployment and the calling of
> >> domain actions etc.
> >>
> >> Cheers
> >>
> >>
> >> Kambiz
> >>
> >>
> >> On 2016-08-03 18:53 CEST, Willie Loyd Tandingan <
> tandingan.wlb@gmail.com>
> >> wrote:
> >>
> >> > Hi Kambiz,
> >> >
> >> > We had a requirement similar to yours. The problem with injecting
> >> > HttpServletRequest to domain services is that it kind of violates the
> >> DDD's
> >> > hexagonal architecture by letting viewer implementation leak into the
> >> > domain layer.
> >> >
> >> > The specific requirement that we had was to be able to pre-upload
> files
> >> > before submitting an action. We developed a custom blob that stores
> an id
> >> > instead of the actual data, a service layer that manages a blob
> storage
> >> > abstracting and running on top of jclouds (we plan to be able to
> support
> >> > other types of storage aside from file system), and a file service
> that
> >> > manages the state of "transient files" (files that were pre-uploaded
> or
> >> > that are returned from actions as result of export functions, etc.) as
> >> well
> >> > as means to persist them (i.e. move them to another
> >> > container/bucket/folder).
> >> >
> >> > We then developed a custom servlet that handles download and upload
> and
> >> > integrate them with isis. This enabled us to implement features like
> >> forced
> >> > attachment content disposition, or on-demand image resizing.
> >> >
> >> > Should you have other ideas or questions, feel free to ask.
> >> >
> >> >
> >> > Best regards,
> >> > Willie
> >> > On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi <da...@m-creations.com>
> >> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I have to implement a legacy file upload service for a customer.
> Files
> >> >> of up to some hundred megabytes are uploaded to a service over HTTP
> POST
> >> >> and a unique ID is returned which can be used to refer to that file,
> add
> >> >> metadata to it etc.
> >> >>
> >> >> Using Blobs is not viable as parsing of the base64 encoded binary
> data
> >> >> and the allocation of multiple strings during request processing
> >> >> increases both the time and space requirements.
> >> >>
> >> >> Ideally, I would be able to create an action on one of my domain
> >> >> services which has a HttpServletRequest argument and take care of
> >> >> reading the binary data and writing them to the file system.
> >> >>
> >> >> I have played around with the request processing part of the
> >> >> restfulobjects viewer/server and managed to get the request/response
> >> >> objects through.
> >> >>
> >> >> I know that this is a real edge case, but I think that it is
> important
> >> >> to have solutions for edge cases, too, if Isis is to be used as a
> >> >> general purpose platform.
> >> >>
> >> >> Could some committers please review the changes and maybe propose a
> >> >> cleaner way of implementing this?
> >> >>
> >> >>
> >> >> https://github.com/m-creations/isis/commit/
> >> aa3b16a5cf463466f5abadbcc8cc73f16857a628
> >> >>
> >> >> Thanks
> >> >>
> >> >>
> >> >> Kambiz
> >> >>
> >> >>
> >>
>
Re: Binary, non-base 64 upload
Posted by Kambiz Darabi <da...@m-creations.com>.
Hello Martin,
On 2016-08-05 13:42 CEST, Martin Grigorov <mg...@apache.org> wrote:
> [...]
> The problem here is that you have to make sure that your code is the very
> first one that reads from the ServletInputStream. Otherwise the body will
> be already consumed.
> You may need to override some Wicket/RESTeasy code to be able to do this
> for the default viewers.
> And this would be much harder than custom Servlet Filter in front of the
> ones by Wicket/RESTeasy
If you look at the change in my original post:
https://github.com/m-creations/isis/commit/aa3b16a5cf463466f5abadbcc8cc73f16857a628
you can see that I added this functionality to the
restfulobjects-viewer, so that I can be sure that the full stream is
handed to the action without any change.
That code is tested against the Isis archetype with this addition to the
SimpleObjectMenu service:
@Action(semantics = SemanticsOf.SAFE)
public void upload(HttpServletRequest request, HttpServletResponse response) {
FileOutputStream out = null;
try {
File outFile = new File("/tmp/request");
out = new FileOutputStream(outFile, false);
IOUtils.copy(request.getInputStream(), out);
out.close();
} catch (IOException e) {
if(out != null) {
try {
out.close();
} catch (IOException e1) {
// LOG something
}
}
// throw something
}
}
and the file in /tmp/request is identical to the posted binary file which is sent by
curl:
curl -X POST --data-binary @big-binary-file
--header "Authorization: Basic c3ZlbjpwYXNz" \
--header "Accept: application/json;profile=urn:org.apache.isis/v1" \
--header "Content-Type: application/octet-stream" \
http://localhost:8080/restful/services/SimpleObjectMenu/actions/upload/invoke
Cheers
Kambiz
>
>> action with the metadata of the files/request, but it would help a lot
>> to have such a facility inside Isis instead of writing a separate
>> servlet and manually integrating its deployment and the calling of
>> domain actions etc.
>>
>> Cheers
>>
>>
>> Kambiz
>>
>>
>> On 2016-08-03 18:53 CEST, Willie Loyd Tandingan <ta...@gmail.com>
>> wrote:
>>
>> > Hi Kambiz,
>> >
>> > We had a requirement similar to yours. The problem with injecting
>> > HttpServletRequest to domain services is that it kind of violates the
>> DDD's
>> > hexagonal architecture by letting viewer implementation leak into the
>> > domain layer.
>> >
>> > The specific requirement that we had was to be able to pre-upload files
>> > before submitting an action. We developed a custom blob that stores an id
>> > instead of the actual data, a service layer that manages a blob storage
>> > abstracting and running on top of jclouds (we plan to be able to support
>> > other types of storage aside from file system), and a file service that
>> > manages the state of "transient files" (files that were pre-uploaded or
>> > that are returned from actions as result of export functions, etc.) as
>> well
>> > as means to persist them (i.e. move them to another
>> > container/bucket/folder).
>> >
>> > We then developed a custom servlet that handles download and upload and
>> > integrate them with isis. This enabled us to implement features like
>> forced
>> > attachment content disposition, or on-demand image resizing.
>> >
>> > Should you have other ideas or questions, feel free to ask.
>> >
>> >
>> > Best regards,
>> > Willie
>> > On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi <da...@m-creations.com>
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> I have to implement a legacy file upload service for a customer. Files
>> >> of up to some hundred megabytes are uploaded to a service over HTTP POST
>> >> and a unique ID is returned which can be used to refer to that file, add
>> >> metadata to it etc.
>> >>
>> >> Using Blobs is not viable as parsing of the base64 encoded binary data
>> >> and the allocation of multiple strings during request processing
>> >> increases both the time and space requirements.
>> >>
>> >> Ideally, I would be able to create an action on one of my domain
>> >> services which has a HttpServletRequest argument and take care of
>> >> reading the binary data and writing them to the file system.
>> >>
>> >> I have played around with the request processing part of the
>> >> restfulobjects viewer/server and managed to get the request/response
>> >> objects through.
>> >>
>> >> I know that this is a real edge case, but I think that it is important
>> >> to have solutions for edge cases, too, if Isis is to be used as a
>> >> general purpose platform.
>> >>
>> >> Could some committers please review the changes and maybe propose a
>> >> cleaner way of implementing this?
>> >>
>> >>
>> >> https://github.com/m-creations/isis/commit/
>> aa3b16a5cf463466f5abadbcc8cc73f16857a628
>> >>
>> >> Thanks
>> >>
>> >>
>> >> Kambiz
>> >>
>> >>
>>
Re: Binary, non-base 64 upload
Posted by Martin Grigorov <mg...@apache.org>.
Hi Kambiz,
On Fri, Aug 5, 2016 at 12:55 PM, Kambiz Darabi <da...@m-creations.com>
wrote:
> Hi,
>
> yes, I fully agree that it violates the DDD idea, but on the other hand,
> I don't see a point in implementing part of the system as a separate
> servlet which doesn't have access to the underlying Isis
> facilities.
>
> We need access to other services to implement business logic
> e.g. depending on the size of the upload or on the file signature (magic
> numbers).
>
> In most cases, we just have to read the InputStream of the servlet and
> write it out to a file, which is much more memory-efficient than reading
> the whole file as a base64 string into memory, decoding it, and then
> calling an action with the Blob.
>
> I'm not opposed to programming some facility (an Isis module?) which
> streams the content of the request to a file - or multiple files in case
> of a multipart request - and _then_ calls an appropriately annotated
>
The problem here is that you have to make sure that your code is the very
first one that reads from the ServletInputStream. Otherwise the body will
be already consumed.
You may need to override some Wicket/RESTeasy code to be able to do this
for the default viewers.
And this would be much harder than custom Servlet Filter in front of the
ones by Wicket/RESTeasy
> action with the metadata of the files/request, but it would help a lot
> to have such a facility inside Isis instead of writing a separate
> servlet and manually integrating its deployment and the calling of
> domain actions etc.
>
> Cheers
>
>
> Kambiz
>
>
> On 2016-08-03 18:53 CEST, Willie Loyd Tandingan <ta...@gmail.com>
> wrote:
>
> > Hi Kambiz,
> >
> > We had a requirement similar to yours. The problem with injecting
> > HttpServletRequest to domain services is that it kind of violates the
> DDD's
> > hexagonal architecture by letting viewer implementation leak into the
> > domain layer.
> >
> > The specific requirement that we had was to be able to pre-upload files
> > before submitting an action. We developed a custom blob that stores an id
> > instead of the actual data, a service layer that manages a blob storage
> > abstracting and running on top of jclouds (we plan to be able to support
> > other types of storage aside from file system), and a file service that
> > manages the state of "transient files" (files that were pre-uploaded or
> > that are returned from actions as result of export functions, etc.) as
> well
> > as means to persist them (i.e. move them to another
> > container/bucket/folder).
> >
> > We then developed a custom servlet that handles download and upload and
> > integrate them with isis. This enabled us to implement features like
> forced
> > attachment content disposition, or on-demand image resizing.
> >
> > Should you have other ideas or questions, feel free to ask.
> >
> >
> > Best regards,
> > Willie
> > On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi <da...@m-creations.com>
> wrote:
> >
> >> Hi,
> >>
> >> I have to implement a legacy file upload service for a customer. Files
> >> of up to some hundred megabytes are uploaded to a service over HTTP POST
> >> and a unique ID is returned which can be used to refer to that file, add
> >> metadata to it etc.
> >>
> >> Using Blobs is not viable as parsing of the base64 encoded binary data
> >> and the allocation of multiple strings during request processing
> >> increases both the time and space requirements.
> >>
> >> Ideally, I would be able to create an action on one of my domain
> >> services which has a HttpServletRequest argument and take care of
> >> reading the binary data and writing them to the file system.
> >>
> >> I have played around with the request processing part of the
> >> restfulobjects viewer/server and managed to get the request/response
> >> objects through.
> >>
> >> I know that this is a real edge case, but I think that it is important
> >> to have solutions for edge cases, too, if Isis is to be used as a
> >> general purpose platform.
> >>
> >> Could some committers please review the changes and maybe propose a
> >> cleaner way of implementing this?
> >>
> >>
> >> https://github.com/m-creations/isis/commit/
> aa3b16a5cf463466f5abadbcc8cc73f16857a628
> >>
> >> Thanks
> >>
> >>
> >> Kambiz
> >>
> >>
>
Re: Binary, non-base 64 upload
Posted by Kambiz Darabi <da...@m-creations.com>.
Hi,
yes, I fully agree that it violates the DDD idea, but on the other hand,
I don't see a point in implementing part of the system as a separate
servlet which doesn't have access to the underlying Isis
facilities.
We need access to other services to implement business logic
e.g. depending on the size of the upload or on the file signature (magic
numbers).
In most cases, we just have to read the InputStream of the servlet and
write it out to a file, which is much more memory-efficient than reading
the whole file as a base64 string into memory, decoding it, and then
calling an action with the Blob.
I'm not opposed to programming some facility (an Isis module?) which
streams the content of the request to a file - or multiple files in case
of a multipart request - and _then_ calls an appropriately annotated
action with the metadata of the files/request, but it would help a lot
to have such a facility inside Isis instead of writing a separate
servlet and manually integrating its deployment and the calling of
domain actions etc.
Cheers
Kambiz
On 2016-08-03 18:53 CEST, Willie Loyd Tandingan <ta...@gmail.com> wrote:
> Hi Kambiz,
>
> We had a requirement similar to yours. The problem with injecting
> HttpServletRequest to domain services is that it kind of violates the DDD's
> hexagonal architecture by letting viewer implementation leak into the
> domain layer.
>
> The specific requirement that we had was to be able to pre-upload files
> before submitting an action. We developed a custom blob that stores an id
> instead of the actual data, a service layer that manages a blob storage
> abstracting and running on top of jclouds (we plan to be able to support
> other types of storage aside from file system), and a file service that
> manages the state of "transient files" (files that were pre-uploaded or
> that are returned from actions as result of export functions, etc.) as well
> as means to persist them (i.e. move them to another
> container/bucket/folder).
>
> We then developed a custom servlet that handles download and upload and
> integrate them with isis. This enabled us to implement features like forced
> attachment content disposition, or on-demand image resizing.
>
> Should you have other ideas or questions, feel free to ask.
>
>
> Best regards,
> Willie
> On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi <da...@m-creations.com> wrote:
>
>> Hi,
>>
>> I have to implement a legacy file upload service for a customer. Files
>> of up to some hundred megabytes are uploaded to a service over HTTP POST
>> and a unique ID is returned which can be used to refer to that file, add
>> metadata to it etc.
>>
>> Using Blobs is not viable as parsing of the base64 encoded binary data
>> and the allocation of multiple strings during request processing
>> increases both the time and space requirements.
>>
>> Ideally, I would be able to create an action on one of my domain
>> services which has a HttpServletRequest argument and take care of
>> reading the binary data and writing them to the file system.
>>
>> I have played around with the request processing part of the
>> restfulobjects viewer/server and managed to get the request/response
>> objects through.
>>
>> I know that this is a real edge case, but I think that it is important
>> to have solutions for edge cases, too, if Isis is to be used as a
>> general purpose platform.
>>
>> Could some committers please review the changes and maybe propose a
>> cleaner way of implementing this?
>>
>>
>> https://github.com/m-creations/isis/commit/aa3b16a5cf463466f5abadbcc8cc73f16857a628
>>
>> Thanks
>>
>>
>> Kambiz
>>
>>
Re: Binary, non-base 64 upload
Posted by Willie Loyd Tandingan <ta...@gmail.com>.
Hi Kambiz,
We had a requirement similar to yours. The problem with injecting
HttpServletRequest to domain services is that it kind of violates the DDD's
hexagonal architecture by letting viewer implementation leak into the
domain layer.
The specific requirement that we had was to be able to pre-upload files
before submitting an action. We developed a custom blob that stores an id
instead of the actual data, a service layer that manages a blob storage
abstracting and running on top of jclouds (we plan to be able to support
other types of storage aside from file system), and a file service that
manages the state of "transient files" (files that were pre-uploaded or
that are returned from actions as result of export functions, etc.) as well
as means to persist them (i.e. move them to another
container/bucket/folder).
We then developed a custom servlet that handles download and upload and
integrate them with isis. This enabled us to implement features like forced
attachment content disposition, or on-demand image resizing.
Should you have other ideas or questions, feel free to ask.
Best regards,
Willie
On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi <da...@m-creations.com> wrote:
> Hi,
>
> I have to implement a legacy file upload service for a customer. Files
> of up to some hundred megabytes are uploaded to a service over HTTP POST
> and a unique ID is returned which can be used to refer to that file, add
> metadata to it etc.
>
> Using Blobs is not viable as parsing of the base64 encoded binary data
> and the allocation of multiple strings during request processing
> increases both the time and space requirements.
>
> Ideally, I would be able to create an action on one of my domain
> services which has a HttpServletRequest argument and take care of
> reading the binary data and writing them to the file system.
>
> I have played around with the request processing part of the
> restfulobjects viewer/server and managed to get the request/response
> objects through.
>
> I know that this is a real edge case, but I think that it is important
> to have solutions for edge cases, too, if Isis is to be used as a
> general purpose platform.
>
> Could some committers please review the changes and maybe propose a
> cleaner way of implementing this?
>
>
> https://github.com/m-creations/isis/commit/aa3b16a5cf463466f5abadbcc8cc73f16857a628
>
> Thanks
>
>
> Kambiz
>
>