Re: jcr2spi spi2dav Sending large data, WorkspaceManager forgets exception

[Angela Schreiber <an...@day.com>]

hi jozef

Jozef Wagner wrote:

> So, should I just set bigger heap for jvm or current spi2dav prototype
> does not support large data? I need to send files as large as 200MB.

you are basically asking whether spi2dav should support large
data, right? so i'd say yes... but i wouldn't be surprised
if it doesn't :).

a possible way improve this would be to make usage of
the global data store (JCR-926). i didn't take a closer look at
it so far... but i figure out that creating and retrieving
the corresponding property item could only use a reference to the
data store, while a separate implementation of the qvaluefactory
could take care of writing/reading to/from the data store.
but that's (up to know) only a very rough draft.

the benefit would obviously be, that reading the property
would be separated from reading the binary value and
we could avoid the xml-bodies transporting binary values.

angela

Re: jcr2spi spi2dav Sending large data

[Jozef Wagner <jo...@gmail.com>]

Hi Angela,

Thank you for your response!

> > I would like to ask about the status of this issue.
> > Is there some plan/roadmap (no matter how vague)? How do you see it in
> > terms of time and resources?
>
> this issue doesn't have high priority for me.
> what i can offer you is:
>
> - quickly hack it in for non-multivalued properties.
> - leave multivalued props for the moment
> - add a patch to the issue (both for spi2dav and jcr-server)
>
> i definitely won't have the time to test it as it
> would be necessary. if you could do that, that would
> help a lot.
>
> would do you think?

That would be great. We don't have large multivalued properties and
our most common node would be of nt:file type.

We would of course test it, as we plan to use it after all :)

Best,
Jozef Wagner

Re: jcr2spi spi2dav Sending large data

[Angela Schreiber <an...@day.com>]

hi jozef

> I would like to ask about the status of this issue.
> Is there some plan/roadmap (no matter how vague)? How do you see it in
> terms of time and resources?

this issue doesn't have high priority for me.
what i can offer you is:

- quickly hack it in for non-multivalued properties.
- leave multivalued props for the moment
- add a patch to the issue (both for spi2dav and jcr-server)

i definitely won't have the time to test it as it
would be necessary. if you could do that, that would
help a lot.

would do you think?
angela

> Sorry if it seems I'm pressing on you, I'm not :). I just want to know
> it in order to be able plan things in my project.

Re: jcr2spi spi2dav Sending large data

[Jozef Wagner <jo...@gmail.com>]

Hi,

I would like to ask about the status of this issue.

Is there some plan/roadmap (no matter how vague)? How do you see it in
terms of time and resources?

Sorry if it seems I'm pressing on you, I'm not :). I just want to know
it in order to be able plan things in my project.

Thank you in advance,
Jozef Wagner

On 1/9/08, Jozef Wagner <jo...@gmail.com> wrote:
> Hi Angela,
>
> On 1/9/08, Angela Schreiber <an...@day.com> wrote:
> >
> > can you open an issue for that?
> >
>
> Done, http://issues.apache.org/jira/browse/JCR-1300
>
> Best,
> Jozef
>

Re: jcr2spi spi2dav Sending large data

[Jozef Wagner <jo...@gmail.com>]

Hi Angela,

On 1/9/08, Angela Schreiber <an...@day.com> wrote:
>
> can you open an issue for that?
>

Done, http://issues.apache.org/jira/browse/JCR-1300

Best,
Jozef

Re: jcr2spi spi2dav Sending large data

[Angela Schreiber <an...@day.com>]

hi jozef

> Well I had to increase maximum heap size for both client and tomcat
> server in order to be able to send 20mb binary property, and it was
> damn slow. I think most of the problem was by this base64 conversion.

probably the easiest (least effort) solution would be
to have an alternative implementation for RepositoryService.addProperty
and RepositoryService.setValue in spi2dav.

an alternative to the global binary store (that consequently
would allow to keep the 'ValueProperty' (that sends the
value array in a webdavish xml-body) would be a straight
forward PUT for single-valued properties. that could (no garantuee)
already work on the server.
for multi-value properties a multipart request could be used.
but that would require some additions on the server.

can you open an issue for that?

regards
angela

Re: jcr2spi spi2dav Sending large data, WorkspaceManager forgets exception

[Jozef Wagner <jo...@gmail.com>]

Hi,

Well I had to increase maximum heap size for both client and tomcat
server in order to be able to send 20mb binary property, and it was
damn slow. I think most of the problem was by this base64 conversion.

That is not good of course and I have to sind a solution pretty soon.

When I use normal webdav interface (/repository/default/ instead of
/server) and upload large file with traditional webdav client (e.g.
bitkinex or cadaver) it is pretty fast and I don't have to increase
maximum heap.

So maybe one quick fix would be to first send the data traditional way
and then modify the node with the help of spi2dav. Maybe this is also
one possibility how to cope with it generally.

Best,
Jozef


On 1/7/08, Angela Schreiber <an...@day.com> wrote:
> Thomas Mueller wrote:
> > Hi,
> >
> >> a possible way improve this would be to make usage of
> >> the global data store (JCR-926)
> >
> > That would be a solution. The idea is to avoid temporary copies of the
> > data, and persist large objects as early as possible. I'm not sure if
> > the data store should be used in the Jackrabbit SPI client
>
> the data store would be on the server. but i would introduce
> means to be able have the binary QValue only 'contain' the
> uri (or some other sort of identifier) and maybe the length.
> the uri would be resolved only if the stream is obtained from
> the value... something like that.
>
> and basically the object could be sent to the server upon
> creating the initial qvalue already... what would be needed
> was a separate qvaluefactory implementation and some extensions
> to the jackrabbit-webapp that would allow to read/write the binary
> objects irrespective of their jcr property.
> that's what i meant my making usage of the global data store.
>
> angela
>

Re: jcr2spi spi2dav Sending large data, WorkspaceManager forgets exception

[Angela Schreiber <an...@day.com>]

Thomas Mueller wrote:
> Hi,
> 
>> a possible way improve this would be to make usage of
>> the global data store (JCR-926)
> 
> That would be a solution. The idea is to avoid temporary copies of the
> data, and persist large objects as early as possible. I'm not sure if
> the data store should be used in the Jackrabbit SPI client

the data store would be on the server. but i would introduce
means to be able have the binary QValue only 'contain' the
uri (or some other sort of identifier) and maybe the length.
the uri would be resolved only if the stream is obtained from
the value... something like that.

and basically the object could be sent to the server upon
creating the initial qvalue already... what would be needed
was a separate qvaluefactory implementation and some extensions
to the jackrabbit-webapp that would allow to read/write the binary
objects irrespective of their jcr property.
that's what i meant my making usage of the global data store.

angela

Re: jcr2spi spi2dav Sending large data, WorkspaceManager forgets exception

[Thomas Mueller <th...@gmail.com>]

Hi,

> a possible way improve this would be to make usage of
> the global data store (JCR-926)

That would be a solution. The idea is to avoid temporary copies of the
data, and persist large objects as early as possible. I'm not sure if
the data store should be used in the Jackrabbit SPI client (it would
introduce a strange dependency). The easiest solution would be to
temporary files on the client side. Another solution is to transfer
large objects in blocks from the Jackrabbit SPI client to the server
(and back).

> we could avoid the xml-bodies transporting binary values.

Yes, base64 conversion should be avoided for performance reasons.

Regards,
Thomas