You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by Sachith Withana <sw...@gmail.com> on 2014/03/05 19:47:15 UTC
File Upload through thrift
Hi all,
I need to transfer files between the Thrift clients and Servers.
There can be large files to be transferred.
Can someone please suggest a way of obtaining this?
--
Thanks,
Sachith Withana
Re: File Upload through thrift
Posted by Rush Manbert <ru...@manbert.com>.
Ah, good to know. Thanks.
On Mar 5, 2014, at 2:40 PM, Ben Craig wrote:
> Don't use list<byte>. Large list performance is pretty bad, because
> elements tend to be serialized one at a time. Use 'binary' or 'string'
> instead. You also get significantly fewer copies this way.
>
> Rush Manbert <ru...@manbert.com> wrote on 03/05/2014 04:31:47 PM:
>
>> From: Rush Manbert <ru...@manbert.com>
>> To: dev@thrift.apache.org,
>> Date: 03/05/2014 04:32 PM
>> Subject: Re: File Upload through thrift
>>
>> Hi Sachith,
>>
>> I would define a thrift struct that has metadata about the file,
>> followed by a member of type list<byte> that stores the file data. I
>> would also define the metadata so it can support transmitting the
>> file in chunks (data that says this is chunk m of n, for instance),
>> where each chunk is sent in a separate message to the server. I
>> would also include some unique cookie value so you can differentiate
>> between two client processes sending the same file.
>>
>> On the client side, figure out how many chunks you need to send the
>> file, then split the file data up between that many instances of
>> your Thrift class. Iterate over the Thrift classes and send each to
>> the server.
>>
>> On the server side, receive the sendFile messages. Use the unique
>> cookie from within the structure to sort the structures you receive
>> into a separate bin for each file you are receiving
>> (std::map<std::string, std::vector<YourThriftStructType *> > would
>> probably work). When you have received all n chunks for a file
>> (which you can tell because of the metadata), then use the collected
>> Thrift structures to reconstitute the file on the server side. The
>> server method can return a value that tells whether or not it thinks
>> the transmission has completed. This approach lets you field send
>> messages from many clients simultaneously with a single server instance.
>>
>> That's my first cut. It probably requires some refinement. ;-)
>>
>> - Rush
>>
>>
>> On Mar 5, 2014, at 1:45 PM, Sachith Withana wrote:
>>
>>> Thanks Henrique and Jens,
>>>
>>> Ideally we'd like to support multiple Gigabytes of data.
>>> As you said, we will potentially end up having a server to do file
>>> uploading and downloading through URLs for Big files.
>>>
>>> But we'd like to support small file (~10MB) uploads using the Thrift
>>> interface at least for now.
>>>
>>> What would be the best approach in achieving that?
>>>
>>>
>>> On Wed, Mar 5, 2014 at 4:32 PM, Jens Geyer <je...@hotmail.com>
> wrote:
>>>
>>>> [...] faster to send a URL and download it separately, [...]
>>>>>
>>>>
>>>> Ack, I was about to write the same.
>>>>
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht----- From: Henrique Mendonça
>>>> Sent: Wednesday, March 5, 2014 10:24 PM
>>>> To: dev@thrift.apache.org
>>>> Subject: Re: File Upload through thrift
>>>>
>>>>
>>>> Hi,
>>>> You can do this, how big are your files? You might find that some
>>>> implementations do have a size limit, not sure about Java, though.
>>>> Anyway, is generally faster to send a URL and download it separately,
>
> if it
>>>> fits you...
>>>>
>>>>
>>>> On 5 March 2014 20:56, Sachith Withana <sw...@gmail.com> wrote:
>>>>
>>>> I'm planning on converting the file into binary type and then do the
>>>>> transfer to the server.
>>>>>
>>>>> What are the possible drawbacks of using this?
>>>>>
>>>>> This feature is for Apache Airavata. We are dealing with multiple
> language
>>>>> clients and a main Java Server.
>>>>> We have to upload ( and download) large files too.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Mar 5, 2014 at 1:47 PM, Sachith Withana
> <sw...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I need to transfer files between the Thrift clients and Servers.
>>>>>> There can be large files to be transferred.
>>>>>>
>>>>>> Can someone please suggest a way of obtaining this?
>>>>>>
>>>>>> --
>>>>>> Thanks,
>>>>>> Sachith Withana
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Sachith Withana
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Sachith Withana
>>
Re: File Upload through thrift
Posted by Ben Craig <be...@apache.org>.
Don't use list<byte>. Large list performance is pretty bad, because
elements tend to be serialized one at a time. Use 'binary' or 'string'
instead. You also get significantly fewer copies this way.
Rush Manbert <ru...@manbert.com> wrote on 03/05/2014 04:31:47 PM:
> From: Rush Manbert <ru...@manbert.com>
> To: dev@thrift.apache.org,
> Date: 03/05/2014 04:32 PM
> Subject: Re: File Upload through thrift
>
> Hi Sachith,
>
> I would define a thrift struct that has metadata about the file,
> followed by a member of type list<byte> that stores the file data. I
> would also define the metadata so it can support transmitting the
> file in chunks (data that says this is chunk m of n, for instance),
> where each chunk is sent in a separate message to the server. I
> would also include some unique cookie value so you can differentiate
> between two client processes sending the same file.
>
> On the client side, figure out how many chunks you need to send the
> file, then split the file data up between that many instances of
> your Thrift class. Iterate over the Thrift classes and send each to
> the server.
>
> On the server side, receive the sendFile messages. Use the unique
> cookie from within the structure to sort the structures you receive
> into a separate bin for each file you are receiving
> (std::map<std::string, std::vector<YourThriftStructType *> > would
> probably work). When you have received all n chunks for a file
> (which you can tell because of the metadata), then use the collected
> Thrift structures to reconstitute the file on the server side. The
> server method can return a value that tells whether or not it thinks
> the transmission has completed. This approach lets you field send
> messages from many clients simultaneously with a single server instance.
>
> That's my first cut. It probably requires some refinement. ;-)
>
> - Rush
>
>
> On Mar 5, 2014, at 1:45 PM, Sachith Withana wrote:
>
> > Thanks Henrique and Jens,
> >
> > Ideally we'd like to support multiple Gigabytes of data.
> > As you said, we will potentially end up having a server to do file
> > uploading and downloading through URLs for Big files.
> >
> > But we'd like to support small file (~10MB) uploads using the Thrift
> > interface at least for now.
> >
> > What would be the best approach in achieving that?
> >
> >
> > On Wed, Mar 5, 2014 at 4:32 PM, Jens Geyer <je...@hotmail.com>
wrote:
> >
> >> [...] faster to send a URL and download it separately, [...]
> >>>
> >>
> >> Ack, I was about to write the same.
> >>
> >>
> >>
> >> -----Ursprüngliche Nachricht----- From: Henrique Mendonça
> >> Sent: Wednesday, March 5, 2014 10:24 PM
> >> To: dev@thrift.apache.org
> >> Subject: Re: File Upload through thrift
> >>
> >>
> >> Hi,
> >> You can do this, how big are your files? You might find that some
> >> implementations do have a size limit, not sure about Java, though.
> >> Anyway, is generally faster to send a URL and download it separately,
if it
> >> fits you...
> >>
> >>
> >> On 5 March 2014 20:56, Sachith Withana <sw...@gmail.com> wrote:
> >>
> >> I'm planning on converting the file into binary type and then do the
> >>> transfer to the server.
> >>>
> >>> What are the possible drawbacks of using this?
> >>>
> >>> This feature is for Apache Airavata. We are dealing with multiple
language
> >>> clients and a main Java Server.
> >>> We have to upload ( and download) large files too.
> >>>
> >>> Thanks.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Mar 5, 2014 at 1:47 PM, Sachith Withana
<sw...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>> I need to transfer files between the Thrift clients and Servers.
> >>>> There can be large files to be transferred.
> >>>>
> >>>> Can someone please suggest a way of obtaining this?
> >>>>
> >>>> --
> >>>> Thanks,
> >>>> Sachith Withana
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Thanks,
> >>> Sachith Withana
> >>>
> >>>
> >>
> >
> >
> > --
> > Thanks,
> > Sachith Withana
>
Re: File Upload through thrift
Posted by Rush Manbert <ru...@manbert.com>.
Hi Sachith,
I would define a thrift struct that has metadata about the file, followed by a member of type list<byte> that stores the file data. I would also define the metadata so it can support transmitting the file in chunks (data that says this is chunk m of n, for instance), where each chunk is sent in a separate message to the server. I would also include some unique cookie value so you can differentiate between two client processes sending the same file.
On the client side, figure out how many chunks you need to send the file, then split the file data up between that many instances of your Thrift class. Iterate over the Thrift classes and send each to the server.
On the server side, receive the sendFile messages. Use the unique cookie from within the structure to sort the structures you receive into a separate bin for each file you are receiving (std::map<std::string, std::vector<YourThriftStructType *> > would probably work). When you have received all n chunks for a file (which you can tell because of the metadata), then use the collected Thrift structures to reconstitute the file on the server side. The server method can return a value that tells whether or not it thinks the transmission has completed. This approach lets you field send messages from many clients simultaneously with a single server instance.
That's my first cut. It probably requires some refinement. ;-)
- Rush
On Mar 5, 2014, at 1:45 PM, Sachith Withana wrote:
> Thanks Henrique and Jens,
>
> Ideally we'd like to support multiple Gigabytes of data.
> As you said, we will potentially end up having a server to do file
> uploading and downloading through URLs for Big files.
>
> But we'd like to support small file (~10MB) uploads using the Thrift
> interface at least for now.
>
> What would be the best approach in achieving that?
>
>
> On Wed, Mar 5, 2014 at 4:32 PM, Jens Geyer <je...@hotmail.com> wrote:
>
>> [...] faster to send a URL and download it separately, [...]
>>>
>>
>> Ack, I was about to write the same.
>>
>>
>>
>> -----Ursprüngliche Nachricht----- From: Henrique Mendonça
>> Sent: Wednesday, March 5, 2014 10:24 PM
>> To: dev@thrift.apache.org
>> Subject: Re: File Upload through thrift
>>
>>
>> Hi,
>> You can do this, how big are your files? You might find that some
>> implementations do have a size limit, not sure about Java, though.
>> Anyway, is generally faster to send a URL and download it separately, if it
>> fits you...
>>
>>
>> On 5 March 2014 20:56, Sachith Withana <sw...@gmail.com> wrote:
>>
>> I'm planning on converting the file into binary type and then do the
>>> transfer to the server.
>>>
>>> What are the possible drawbacks of using this?
>>>
>>> This feature is for Apache Airavata. We are dealing with multiple language
>>> clients and a main Java Server.
>>> We have to upload ( and download) large files too.
>>>
>>> Thanks.
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Mar 5, 2014 at 1:47 PM, Sachith Withana <sw...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I need to transfer files between the Thrift clients and Servers.
>>>> There can be large files to be transferred.
>>>>
>>>> Can someone please suggest a way of obtaining this?
>>>>
>>>> --
>>>> Thanks,
>>>> Sachith Withana
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Sachith Withana
>>>
>>>
>>
>
>
> --
> Thanks,
> Sachith Withana
Re: File Upload through thrift
Posted by Sachith Withana <sw...@gmail.com>.
Thanks Henrique and Jens,
Ideally we'd like to support multiple Gigabytes of data.
As you said, we will potentially end up having a server to do file
uploading and downloading through URLs for Big files.
But we'd like to support small file (~10MB) uploads using the Thrift
interface at least for now.
What would be the best approach in achieving that?
On Wed, Mar 5, 2014 at 4:32 PM, Jens Geyer <je...@hotmail.com> wrote:
> [...] faster to send a URL and download it separately, [...]
>>
>
> Ack, I was about to write the same.
>
>
>
> -----Ursprüngliche Nachricht----- From: Henrique Mendonça
> Sent: Wednesday, March 5, 2014 10:24 PM
> To: dev@thrift.apache.org
> Subject: Re: File Upload through thrift
>
>
> Hi,
> You can do this, how big are your files? You might find that some
> implementations do have a size limit, not sure about Java, though.
> Anyway, is generally faster to send a URL and download it separately, if it
> fits you...
>
>
> On 5 March 2014 20:56, Sachith Withana <sw...@gmail.com> wrote:
>
> I'm planning on converting the file into binary type and then do the
>> transfer to the server.
>>
>> What are the possible drawbacks of using this?
>>
>> This feature is for Apache Airavata. We are dealing with multiple language
>> clients and a main Java Server.
>> We have to upload ( and download) large files too.
>>
>> Thanks.
>>
>>
>>
>>
>>
>> On Wed, Mar 5, 2014 at 1:47 PM, Sachith Withana <sw...@gmail.com>
>> wrote:
>>
>> > Hi all,
>> >
>> > I need to transfer files between the Thrift clients and Servers.
>> > There can be large files to be transferred.
>> >
>> > Can someone please suggest a way of obtaining this?
>> >
>> > --
>> > Thanks,
>> > Sachith Withana
>> >
>> >
>>
>>
>> --
>> Thanks,
>> Sachith Withana
>>
>>
>
--
Thanks,
Sachith Withana
Re: File Upload through thrift
Posted by Jens Geyer <je...@hotmail.com>.
> [...] faster to send a URL and download it separately, [...]
Ack, I was about to write the same.
-----Ursprüngliche Nachricht-----
From: Henrique Mendonça
Sent: Wednesday, March 5, 2014 10:24 PM
To: dev@thrift.apache.org
Subject: Re: File Upload through thrift
Hi,
You can do this, how big are your files? You might find that some
implementations do have a size limit, not sure about Java, though.
Anyway, is generally faster to send a URL and download it separately, if it
fits you...
On 5 March 2014 20:56, Sachith Withana <sw...@gmail.com> wrote:
> I'm planning on converting the file into binary type and then do the
> transfer to the server.
>
> What are the possible drawbacks of using this?
>
> This feature is for Apache Airavata. We are dealing with multiple language
> clients and a main Java Server.
> We have to upload ( and download) large files too.
>
> Thanks.
>
>
>
>
>
> On Wed, Mar 5, 2014 at 1:47 PM, Sachith Withana <sw...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I need to transfer files between the Thrift clients and Servers.
> > There can be large files to be transferred.
> >
> > Can someone please suggest a way of obtaining this?
> >
> > --
> > Thanks,
> > Sachith Withana
> >
> >
>
>
> --
> Thanks,
> Sachith Withana
>
Re: File Upload through thrift
Posted by Henrique Mendonça <he...@apache.org>.
Hi,
You can do this, how big are your files? You might find that some
implementations do have a size limit, not sure about Java, though.
Anyway, is generally faster to send a URL and download it separately, if it
fits you...
On 5 March 2014 20:56, Sachith Withana <sw...@gmail.com> wrote:
> I'm planning on converting the file into binary type and then do the
> transfer to the server.
>
> What are the possible drawbacks of using this?
>
> This feature is for Apache Airavata. We are dealing with multiple language
> clients and a main Java Server.
> We have to upload ( and download) large files too.
>
> Thanks.
>
>
>
>
>
> On Wed, Mar 5, 2014 at 1:47 PM, Sachith Withana <sw...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I need to transfer files between the Thrift clients and Servers.
> > There can be large files to be transferred.
> >
> > Can someone please suggest a way of obtaining this?
> >
> > --
> > Thanks,
> > Sachith Withana
> >
> >
>
>
> --
> Thanks,
> Sachith Withana
>
Re: File Upload through thrift
Posted by Sachith Withana <sw...@gmail.com>.
I'm planning on converting the file into binary type and then do the
transfer to the server.
What are the possible drawbacks of using this?
This feature is for Apache Airavata. We are dealing with multiple language
clients and a main Java Server.
We have to upload ( and download) large files too.
Thanks.
On Wed, Mar 5, 2014 at 1:47 PM, Sachith Withana <sw...@gmail.com> wrote:
> Hi all,
>
> I need to transfer files between the Thrift clients and Servers.
> There can be large files to be transferred.
>
> Can someone please suggest a way of obtaining this?
>
> --
> Thanks,
> Sachith Withana
>
>
--
Thanks,
Sachith Withana