Byte Streaming

[Alain K <aw...@yahoo.fr>]

Hello,

Is it possible to send large data via xmlrpc without
loading everything into memory first using some kind
of streaming mecanism.

Thanks for your help.

Alain.

___________________________________________________________
Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en français !
Yahoo! Mail : http://fr.mail.yahoo.com

Re: Byte Streaming

[Daniel Rall <dl...@finemaltcoding.com>]

"John Wilson" <tu...@wilson.co.uk> writes:

> ----- Original Message -----
> From: "Alain K" <aw...@yahoo.fr>
> To: <rp...@xml.apache.org>
> Sent: Wednesday, March 06, 2002 4:44 PM
> Subject: Byte Streaming
>
>
>> Hello,
>>
>> Is it possible to send large data via xmlrpc without
>> loading everything into memory first using some kind
>> of streaming mecanism.
>
> Because the XML-RPC spec requires a correct content-length header then you
> really have to assemble the message in memory to work out how long it is
> before sending it.
>
> If you are doing something like sending the Base64 encoded contents of a
> file you could avoid this because you know the file length and can work out
> the length of the message before reading the file. However this is special
> case stuff and you would not really expect a general purpose XML-RPC
> implementation to do it.

There's also been discussion on this sort of thing on the rpc-dev
list.  You might want to search the archive and read it over...

Re: Byte Streaming

[Daniel Rall <dl...@finemaltcoding.com>]

"John Wilson" <tu...@wilson.co.uk> writes:

> ----- Original Message -----
> From: "Alain K" <aw...@yahoo.fr>
> To: <rp...@xml.apache.org>
> Sent: Wednesday, March 06, 2002 4:44 PM
> Subject: Byte Streaming
>
>
>> Hello,
>>
>> Is it possible to send large data via xmlrpc without
>> loading everything into memory first using some kind
>> of streaming mecanism.
>
> Because the XML-RPC spec requires a correct content-length header then you
> really have to assemble the message in memory to work out how long it is
> before sending it.
>
> If you are doing something like sending the Base64 encoded contents of a
> file you could avoid this because you know the file length and can work out
> the length of the message before reading the file. However this is special
> case stuff and you would not really expect a general purpose XML-RPC
> implementation to do it.

There's also been discussion on this sort of thing on the rpc-dev
list.  You might want to search the archive and read it over...

Re: Byte Streaming

[John Wilson <tu...@wilson.co.uk>]

----- Original Message -----
From: "Alain K" <aw...@yahoo.fr>
To: <rp...@xml.apache.org>
Sent: Wednesday, March 06, 2002 4:44 PM
Subject: Byte Streaming


> Hello,
>
> Is it possible to send large data via xmlrpc without
> loading everything into memory first using some kind
> of streaming mecanism.

Because the XML-RPC spec requires a correct content-length header then you
really have to assemble the message in memory to work out how long it is
before sending it.

If you are doing something like sending the Base64 encoded contents of a
file you could avoid this because you know the file length and can work out
the length of the message before reading the file. However this is special
case stuff and you would not really expect a general purpose XML-RPC
implementation to do it.

John Wilson
The Wilson Partnership
http://www.wilson.co.uk

Re: Byte Streaming

[John Wilson <tu...@wilson.co.uk>]

----- Original Message -----
From: "Alain K" <aw...@yahoo.fr>
To: <rp...@xml.apache.org>
Sent: Wednesday, March 06, 2002 4:44 PM
Subject: Byte Streaming


> Hello,
>
> Is it possible to send large data via xmlrpc without
> loading everything into memory first using some kind
> of streaming mecanism.

Because the XML-RPC spec requires a correct content-length header then you
really have to assemble the message in memory to work out how long it is
before sending it.

If you are doing something like sending the Base64 encoded contents of a
file you could avoid this because you know the file length and can work out
the length of the message before reading the file. However this is special
case stuff and you would not really expect a general purpose XML-RPC
implementation to do it.

John Wilson
The Wilson Partnership
http://www.wilson.co.uk

Re: Byte Streaming

[Paul Libbrecht <pa...@activemath.org>]

But simple http CAN stream !
http 1.1 even can recover broken streams.
Just... xml-rpc cannot.
Because you can do it on top (and do recovery as well).

Paul


On Jeudi, mars 7, 2002, at 12:56 , ward harold wrote:

> If you must use HTTP then building streaming over
> XML-RPC as Paul suggests is an appropriate thing to
> do. If HTTP isn't necessarily mandated then you should
> investigate BEEP, www.beepcore.org, as a mechanism for
> defining a protocol that meets your application's bulk
> data transfer requirements.
>
> ... WkH

Re: Byte Streaming

[Paul Libbrecht <pa...@activemath.org>]

But simple http CAN stream !
http 1.1 even can recover broken streams.
Just... xml-rpc cannot.
Because you can do it on top (and do recovery as well).

Paul


On Jeudi, mars 7, 2002, at 12:56 , ward harold wrote:

> If you must use HTTP then building streaming over
> XML-RPC as Paul suggests is an appropriate thing to
> do. If HTTP isn't necessarily mandated then you should
> investigate BEEP, www.beepcore.org, as a mechanism for
> defining a protocol that meets your application's bulk
> data transfer requirements.
>
> ... WkH

Re: Byte Streaming

[ward harold <wa...@yahoo.com>]

If you must use HTTP then building streaming over
XML-RPC as Paul suggests is an appropriate thing to
do. If HTTP isn't necessarily mandated then you should
investigate BEEP, www.beepcore.org, as a mechanism for
defining a protocol that meets your application's bulk
data transfer requirements.

... WkH

--- Paul Libbrecht <pa...@activemath.org> wrote:
> No, no, no and no !
> (I should be make some nuances though)
> 
> The XML-RPC specification experessedly specify that
> the Content-Length 
> of the XML-RPC message is part of the header so,
> unless you are doing it 
> in special cases where you would know the size in
> advance (and would 
> then need to modify the current library), it is not
> possible.
> 
> Stop, don't be scandalized like I was the first time
> I read this.
> There's half a ton of good reasons you do want to
> know this size in 
> advance.
> For example, to allocate buffers for easier parsing.
> But also to master 
> resources in general. Still looks like a pity ??
> Second you would need the complete API to be changed
> to support 
> streaming progress monitoring, so the great simple
> execute(String 
> methodName, Vector params) would no more be true
> (and the even easier 
> direct calls to an object would also be impossible).
> You would need some 
> delicate objects instead.
> XML-RPC is simple and that makes its strength !!
> (simply compare the number of implementations for
> XML-RPC to the ones 
> for SOAP)
> 
> Well... why not, simply build streaming ON TOP of
> XML-RPC ??
> That is the way to go, believe me.
> You request a URL to start sending your chunks.
> You then send each of your chunks to this URL (which
> presumably calls 
> another method that makes the storage, processing
> and monitoring of the 
> calls) and call the close method.
> Advantages ? If only half of the data is through, it
> may be usable. 
> Better resource control and...
> 
> This should definitely be in some FAQ.
> This together with an answer to the question "Why
> can't I put some XML 
> in the middle of an XML-RPC call if not as a string
> ?".
> 
> Paul
> 
> 
> 
> On Mercredi, mars 6, 2002, at 05:44 , Alain K wrote:
> 
> > Hello,
> >
> > Is it possible to send large data via xmlrpc
> without
> > loading everything into memory first using some
> kind
> > of streaming mecanism.
> >
> > Thanks for your help.
> >
> > Alain.
> >
> >
>
___________________________________________________________
> > Do You Yahoo!? -- Une adresse @yahoo.fr gratuite
> et en fran�ais !
> > Yahoo! Mail : http://fr.mail.yahoo.com
> >
> 


__________________________________________________
Do You Yahoo!?
Try FREE Yahoo! Mail - the world's greatest free email!
http://mail.yahoo.com/

Re: Byte Streaming

[ward harold <wa...@yahoo.com>]

If you must use HTTP then building streaming over
XML-RPC as Paul suggests is an appropriate thing to
do. If HTTP isn't necessarily mandated then you should
investigate BEEP, www.beepcore.org, as a mechanism for
defining a protocol that meets your application's bulk
data transfer requirements.

... WkH

--- Paul Libbrecht <pa...@activemath.org> wrote:
> No, no, no and no !
> (I should be make some nuances though)
> 
> The XML-RPC specification experessedly specify that
> the Content-Length 
> of the XML-RPC message is part of the header so,
> unless you are doing it 
> in special cases where you would know the size in
> advance (and would 
> then need to modify the current library), it is not
> possible.
> 
> Stop, don't be scandalized like I was the first time
> I read this.
> There's half a ton of good reasons you do want to
> know this size in 
> advance.
> For example, to allocate buffers for easier parsing.
> But also to master 
> resources in general. Still looks like a pity ??
> Second you would need the complete API to be changed
> to support 
> streaming progress monitoring, so the great simple
> execute(String 
> methodName, Vector params) would no more be true
> (and the even easier 
> direct calls to an object would also be impossible).
> You would need some 
> delicate objects instead.
> XML-RPC is simple and that makes its strength !!
> (simply compare the number of implementations for
> XML-RPC to the ones 
> for SOAP)
> 
> Well... why not, simply build streaming ON TOP of
> XML-RPC ??
> That is the way to go, believe me.
> You request a URL to start sending your chunks.
> You then send each of your chunks to this URL (which
> presumably calls 
> another method that makes the storage, processing
> and monitoring of the 
> calls) and call the close method.
> Advantages ? If only half of the data is through, it
> may be usable. 
> Better resource control and...
> 
> This should definitely be in some FAQ.
> This together with an answer to the question "Why
> can't I put some XML 
> in the middle of an XML-RPC call if not as a string
> ?".
> 
> Paul
> 
> 
> 
> On Mercredi, mars 6, 2002, at 05:44 , Alain K wrote:
> 
> > Hello,
> >
> > Is it possible to send large data via xmlrpc
> without
> > loading everything into memory first using some
> kind
> > of streaming mecanism.
> >
> > Thanks for your help.
> >
> > Alain.
> >
> >
>
___________________________________________________________
> > Do You Yahoo!? -- Une adresse @yahoo.fr gratuite
> et en fran�ais !
> > Yahoo! Mail : http://fr.mail.yahoo.com
> >
> 


__________________________________________________
Do You Yahoo!?
Try FREE Yahoo! Mail - the world's greatest free email!
http://mail.yahoo.com/

Re: Byte Streaming

[Paul Libbrecht <pa...@activemath.org>]

No, no, no and no !
(I should be make some nuances though)

The XML-RPC specification experessedly specify that the Content-Length 
of the XML-RPC message is part of the header so, unless you are doing it 
in special cases where you would know the size in advance (and would 
then need to modify the current library), it is not possible.

Stop, don't be scandalized like I was the first time I read this.
There's half a ton of good reasons you do want to know this size in 
advance.
For example, to allocate buffers for easier parsing. But also to master 
resources in general. Still looks like a pity ??
Second you would need the complete API to be changed to support 
streaming progress monitoring, so the great simple execute(String 
methodName, Vector params) would no more be true (and the even easier 
direct calls to an object would also be impossible). You would need some 
delicate objects instead.
XML-RPC is simple and that makes its strength !!
(simply compare the number of implementations for XML-RPC to the ones 
for SOAP)

Well... why not, simply build streaming ON TOP of XML-RPC ??
That is the way to go, believe me.
You request a URL to start sending your chunks.
You then send each of your chunks to this URL (which presumably calls 
another method that makes the storage, processing and monitoring of the 
calls) and call the close method.
Advantages ? If only half of the data is through, it may be usable. 
Better resource control and...

This should definitely be in some FAQ.
This together with an answer to the question "Why can't I put some XML 
in the middle of an XML-RPC call if not as a string ?".

Paul



On Mercredi, mars 6, 2002, at 05:44 , Alain K wrote:

> Hello,
>
> Is it possible to send large data via xmlrpc without
> loading everything into memory first using some kind
> of streaming mecanism.
>
> Thanks for your help.
>
> Alain.
>
> ___________________________________________________________
> Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en français !
> Yahoo! Mail : http://fr.mail.yahoo.com
>

Re: Byte Streaming

[Paul Libbrecht <pa...@activemath.org>]

No, no, no and no !
(I should be make some nuances though)

The XML-RPC specification experessedly specify that the Content-Length 
of the XML-RPC message is part of the header so, unless you are doing it 
in special cases where you would know the size in advance (and would 
then need to modify the current library), it is not possible.

Stop, don't be scandalized like I was the first time I read this.
There's half a ton of good reasons you do want to know this size in 
advance.
For example, to allocate buffers for easier parsing. But also to master 
resources in general. Still looks like a pity ??
Second you would need the complete API to be changed to support 
streaming progress monitoring, so the great simple execute(String 
methodName, Vector params) would no more be true (and the even easier 
direct calls to an object would also be impossible). You would need some 
delicate objects instead.
XML-RPC is simple and that makes its strength !!
(simply compare the number of implementations for XML-RPC to the ones 
for SOAP)

Well... why not, simply build streaming ON TOP of XML-RPC ??
That is the way to go, believe me.
You request a URL to start sending your chunks.
You then send each of your chunks to this URL (which presumably calls 
another method that makes the storage, processing and monitoring of the 
calls) and call the close method.
Advantages ? If only half of the data is through, it may be usable. 
Better resource control and...

This should definitely be in some FAQ.
This together with an answer to the question "Why can't I put some XML 
in the middle of an XML-RPC call if not as a string ?".

Paul



On Mercredi, mars 6, 2002, at 05:44 , Alain K wrote:

> Hello,
>
> Is it possible to send large data via xmlrpc without
> loading everything into memory first using some kind
> of streaming mecanism.
>
> Thanks for your help.
>
> Alain.
>
> ___________________________________________________________
> Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en français !
> Yahoo! Mail : http://fr.mail.yahoo.com
>