You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-dev@axis.apache.org by Kapila Dissanayake <ka...@gmail.com> on 2006/05/13 07:14:59 UTC
Re: [AXIS2] MTOM performance Measuring - http_transport_utils.c modified
Hi ,
I did another modification to improve the messege transfer speed.
http_transport_utils.c was modified. (Buffer size was changed from 1024 to
1024 * 1024)
Now the transfer speed has been changed significantly. Please see the table
below.
The attached file includes the svn diff of the change.
Thank You
Kapila
(File Size) vs (Time Taken in milli seconds) to send binary
attachments Environment ---------- Processor -
Intel Celeron M 1.40GHz RAM - 256 MB OS - Fedora Core 4.
Java
Server - axis2 Java C Server - axis2 mtom C and java test
clients were used http_transport_utils.c was modified. (Buffer size
was changed from 1024 to 1024 * 1024) Time Taken in milli seconds
File
Size (kb) Java Server Java Server C Server C Server C Server C
Server C Server C Server C Server C Server Java Client (ms) C
Client (ms) C Client (ms) Java Client (ms)
*data_handler.c
Buffer Size 1024* *data_handler.c 'Buffer Size 1024 X 1024* *Using
stat* *http_transport_utils.c
'Buffer Size 1024 X 1024* * * *Buffer Size '1024* *Buffer Size '1024 X 1024*
*Using stat* *http_transport_utils.c 'Buffer Size 1024 X 1024*
----------------- ----------------- ----------------- -----------------
----------------- ----------------- ----------------- -----------------
----------------- --------------------------------- -----------------
----------------- ----------------- -----------------
--------------------------------- 13.8 147 158 73 1884 78 76 188 143
136 136 28.2 515 488 78 1912 78 78 222 413 364 387 37.8 538 694
131 1917 81 81 232 474 377 396 132 571 807 212 2177 102 83 263
420 381 432 154 547 672 239 2215 117 84 347 389 383 517 165 688
683 397 2380 205 85 364 400 393 602 561 1340 1024 995 2537 777 100
637 899 775 766 762 1766 1336 1903 2890 1458 111 1248 1218 1051
1055 1162 3464 1970 4859 3602 3405 135 1702 1718 1712 1092 1761
4666 3274 10928 8028 7927 188 2780 3249 2691 2596 2081 6777 2993
15066 11008 10864 219 3595 3281 2859 3185 4241 23038 Could not
transfer (Java client Broken) 52853 33351 32957
519 16234 115777 7991 6499
On 5/12/06, Samisa Abeysinghe <sa...@gmail.com> wrote:
>
> BTW,
> If we are going to use chunking at some point in time, then the data
> handler logic for reading file too has to change to keep track of the
> amount of chunks already read, and current position of read pointer.
> There is no provision for such reading in the code as of now, neither in
> the data handler nor in the mime_output that uses the data handler API.
> So there is considerable amount of work on the part of the data hander
> and the places that uses the API of data handler to read to turn on
> chunking as far as I can see.
> Thanks,
> Samisa...
>
> Samisa Abeysinghe wrote:
>
> > Nabeel wrote:
> >
> >> Kapila,
> >> With your latest changes you always read the file in one go
> >> (i.e. not in chunks) and you don't need a loop. The attached compact
> >> code does the same thing.
> >>
> >> Suggestion:
> >> When we need to attach *very* large file, we may not be keep the
> >> whole file in memory. IMO, this where chunk reading is really useful.
> >> If you are to implement this, you'll have to change the code from
> >> data_handler -> mime_body_part -> mime_output to om_output so that
> >> you'll keep only chunk_size amount of bytes in memory at any given
> time.
> >
> >
> > Chunking is a very good idea, in fact that is the perfect solution for
> > large attachments.
> > However, AFAIK, we have chunking capability only on the client side
> > transport, not on the server side transport.
> > Hence, we got to implement chunking capability into simple axis server
> > transport first. (May be we can raise a Jira on this). Or we have to
> > see if we can piggyback on Apache for chunking.
> > Secondly, we got to research and find out the ideal chunk size. As we
> > have seen through the results sent by Kapila, too small a chunk size
> > makes too large file transmission very slow and too large chunk size
> > makes small file transmission very slow. Best is to not use chunking
> > for small files and use chunking, with a reasonable chunk size, for
> > large files. So we have several parameters to figure out.
> > i. Max size we can read a file in one go without chunking.
> > ii. When chunking is turned on, ideal chunk size.
> > As I mentioned earlier, we need some research around this to decide on
> > the figures.
> >
> > Thanks,
> > Samisa...
> >
> >>
> >> -Nabeel
> >>
> >> Samisa Abeysinghe wrote:
> >>
> >>> Cool, so there sure is an improvement when using stat on read side
> >>> of data handler. Hence lets use this solution.
> >>> Still we need to figure out what goes wrong with large files beyond 1M
> >>>
> >>> Samisa...
> >>>
> >>> Kapila Dissanayake wrote:
> >>>
> >>>> Hi
> >>>>
> >>>> I did some modification to the data handler. Insted of using fixed
> >>>> buffer size to read files
> >>>> 'stat' is used.
> >>>>
> >>>> Please see the perfomance details.
> >>>>
> >>>> Thank You
> >>>>
> >>>> Kapila
> >>>>
> >>>>
> >>>> Environment ---------- Processor - Intel
> >>>> Celeron M 1.40GHz
> >>>> RAM - 256 MB OS - Fedora Core 4. Java Server -
> >>>> axis2 Java C Server - axis2 mtom C and java test
> >>>> clients were used (in axis2 c data_handler.c, different buffer
> >>>> sizes were used to read files)
> >>>> Time Taken in milli seconds
> >>>> File Size (kb) Java Server (ms) Java Server
> >>>> (ms) C Server (ms) C Server (ms)
> >>>> Java Client (ms) C Client (ms) C
> >>>> Client (ms) Java Client (ms)
> >>>> *Buffer Size 1024 * *Buffer
> >>>> Size 1024 X 1024 * *Using stat* * * *Buffer Size '1024
> >>>> * *Buffer Size '1024 X 1024 * *Using stat *
> >>>> ----------------- ----------------- -----------------
> >>>> ----------------- ----------------- -----------------
> >>>> ----------------- ----------------- -----------------
> >>>> ----------------- ----------------- -----------------
> >>>> -----------------
> >>>> 13.8 147 158 73 1884
> >>>> 78 188 143 136
> >>>> 28.2 515 488 78 1912
> >>>> 78 222 413 364
> >>>> 37.8 538 694 131 1917
> >>>> 81 232 474 377
> >>>> 132 571 807 212 2177
> >>>> 102 263 420 381
> >>>> 154 547 672 239 2215
> >>>> 117 347 389 383
> >>>> 165 688 683 397 2380
> >>>> 205 364 400 393
> >>>> 561 1340 1024 995 2537
> >>>> 777 637 899 775
> >>>> 762 1766 1336 1903 2890
> >>>> 1458 1248 1218 1051
> >>>> 1162 3464 1970 4859 3602
> >>>> 3405 1702 1718 1712
> >>>> 1761 4666 3274 10928 8028
> >>>> 7927 2780 3249 2691
> >>>> 2081 6777 2993 15066 11008
> >>>> 10864 3595 3281 2859
> >>>> 4241 23038 Could not transfer (Java client
> >>>> Broken) 52853 33351 32957 16234
> >>>> 115777 7991
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 5/10/06, *Samisa Abeysinghe* <samisa.abeysinghe@gmail.com
> >>>> <ma...@gmail.com>> wrote:
> >>>>
> >>>> Well, from this I am certain that we should not change the
> >>>> buffer size
> >>>> to 1024*1024, rather keep it at 1024.
> >>>> Please try with stat, using a dynamic buffer size based on the
> >>>> file
> >>>> size, my gut feel is that it is going to be the best solution.
> >>>>
> >>>> Thanks,
> >>>> Samisa...
> >>>>
> >>>> Kapila Dissanayake wrote:
> >>>>
> >>>> >
> >>>> > Hi,
> >>>> >
> >>>> > I did some testing agian with to measure the binary file
> >>>> transfer
> >>>> > times. In axis2 c data_handler.c, different buffer sizes
> >>>> (1024 and
> >>>> > 1024*1024) were used to read files.
> >>>> >
> >>>> > I think using fixed size buffers to read binary files in
> axis2-C
> >>>> has
> >>>> > been affected in some cases to slow down the transfer.
> >>>> >
> >>>> > Thanks
> >>>> >
> >>>> > Kapila
> >>>> >
> >>>> > Please see the following test results
> >>>> >
> >>>> > Environment
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > ----------
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > Processor - Intel Celeron M 1.40GHz
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > RAM - 256 MB
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > OS - Fedora Core 4.
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > Java Server - axis2 Java
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > C Server - axis2
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > mtom C and java test clients were used
> >>>> >
> >>>> >
> >>>> >
> >>>> > Time Taken in milli seconds
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > File Size (kb) Java Server (ms) Java
> >>>> Server (ms) C
> >>>> > Server (ms) C Server (ms)
> >>>> > Java Client (ms) C
> >>>> Client (ms) C Client (ms)
> >>>> Java
> >>>> > Client (ms)
> >>>> > *Buffer Size*
> >>>> *1024 * *1024 X 1024* * * *1024*
> >>>> > *1024 X 1024 *
> >>>> > ----------------- ----------------- -----------------
> >>>> > ----------------- ----------------- -----------------
> >>>> > ----------------- ----------------- -----------------
> >>>> > ----------------- ----------------- -----------------
> >>>> > 13.8 147 158
> >>>> 73 1884 188 143
> >>>> > 28.2 515 488
> >>>> 78 1912 222 413
> >>>> > 37.8 538 694 131
> >>>> 1917 232 474
> >>>> > 132 571 807 212
> >>>> 2177 263 420
> >>>> > 154 547 672 239
> >>>> 2215 347 389
> >>>> > 165 688 683 397
> >>>> 2380 364 400
> >>>> > 561 1340 1024 995
> >>>> 2537 637 899
> >>>> > 762 1766 1336
> >>>> 1903 2890 1248 1218
> >>>> >
> >>>> 1162 3464 1970 4859
> >>>> 3602 1702 1718
> >>>>
> >>>> > 1761 4666 3274 10928
> >>>> 8028 2780 3249
> >>>> > 2081 6777 2993 15066
> >>>> 11008 3595 3281
> >>>> > 4241 23038 Could not transfer (Java client
> >>>> Broken)
> >>>> > 52853 33351 16234 115777
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > On 5/8/06, *Samisa Abeysinghe* <samisa@wso2.com
> >>>> <ma...@wso2.com>
> >>>> > <mailto: samisa@wso2.com <ma...@wso2.com>>> wrote:
> >>>> >
> >>>> > Hmmm, Interesting. There seem to be a problem in C client
> in
> >>>> case of
> >>>> > large attachments.
> >>>> > This must be due to the fact that we are using a loop to
> >>>> read the
> >>>> > file.
> >>>> > If we stat and read the file at once, there could be an
> >>>> improvement.
> >>>> >
> >>>> > Why is it so slow, in C vs C case? (Third column) This is
> >>>> worth
> >>>> > investigating.
> >>>> >
> >>>> > Thanks,
> >>>> > Samisa...
> >>>> >
> >>>> > Kapila Dissanayake wrote:
> >>>> >
> >>>> > > Hi,
> >>>> > >
> >>>> > > I did some testing on binary attachment transfer times,
> >>>> with axis2
> >>>> > > 'java' and 'C' servers with sample mtom clients. It took
> >>>> very high
> >>>> > > transfer times for large size attachments. Some minor
> >>>> modificaions
> >>>> > > were done to the data_handler.c and it was sent to the
> >>>> mailing list.
> >>>> > >
> >>>> > >
> >>>> > > (File Size) vs (Time Taken in milli seconds) to transfer
> >>>> binary
> >>>> > > attachments
> >>>> > >
> >>>> > > Environment
> >>>> > > ----------
> >>>> > > Processor - Intel Celeron M 1.40GHz
> >>>> > > RAM - 256 MB
> >>>> > > OS - Fedora Core 4.
> >>>> > > Java Server - axis2 Java
> >>>> > > C Server - axis2
> >>>> > > mtom C and java test clients were used
> >>>> > > Time Taken in milli seconds
> >>>> > >
> >>>> > > File Size (kb)
> >>>> > >
> >>>> > > Java Server
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > > Java Server
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > > C Server
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > > C Server
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Java Client
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > > C Client
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > > C Client
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > > Java Client
> >>>> > >
> >>>> > > (ms)
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > > 28
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 445
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 274
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2004
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 1009
> >>>> > >
> >>>> > >
> >>>> > > 38
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 617
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 297
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 1957
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 842
> >>>> > >
> >>>> > >
> >>>> > > 132
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 593
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 527
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2181
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 726
> >>>> > >
> >>>> > >
> >>>> > > 154
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 551
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 447
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2081
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 677
> >>>> > >
> >>>> > >
> >>>> > > 165
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 440
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 743
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2147
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 676
> >>>> > >
> >>>> > >
> >>>> > > 561
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 1029
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 418
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2581
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 1075
> >>>> > >
> >>>> > >
> >>>> > > 762
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2137
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 1572
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 3231
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2356
> >>>> > >
> >>>> > >
> >>>> > > 1162
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 3172
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2962
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 3981
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 3100
> >>>> > >
> >>>> > >
> >>>> > > 1761
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 7672
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 5494
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 8186
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 3041
> >>>> > >
> >>>> > >
> >>>> > > 2081
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 7768
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 6837
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 11757
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 3499
> >>>> > >
> >>>> > >
> >>>> > > 4241
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 14465
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 24060
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 36168
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Could not transfer (Error Occured in Java client)
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Thank You
> >>>> > >
> >>>> > > Kapila
> >>>> >
> >>>> >
> >>>> >
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> ------------------------------------------------------------------------
> >>
> >> #include <stdio.h>
> >> #include <sys/stat.h>
> >>
> >> int read_from_file(char *file_name, char** output_stream, int
> >> *output_stream_size);
> >>
> >> int main(int argc, char **argv)
> >> {
> >> int size;
> >> char *output;
> >>
> >> if (argc != 2)
> >> {
> >> printf("usage: read_binary <file_name>\n");
> >> return 0;
> >> }
> >> if (read_from_file(argv[1], &output, &size))
> >> printf("read successful\n");
> >> if (output)
> >> free(output);
> >> return 1;
> >> }
> >>
> >> int read_from_file(char *file_name, char** output_stream, int
> >> *output_stream_size)
> >> {
> >> FILE *f = NULL;
> >> struct stat stat_p;
> >> int count = 0;
> >>
> >> if (!file_name)
> >> {
> >> printf("error: file not given \n");
> >> return 0;
> >> }
> >> f = fopen(file_name, "rb");
> >> if (!f)
> >> {
> >> printf("error:cannot open file %s\n", file_name);
> >> return ;
> >> }
> >> if ( -1 == stat (file_name, &stat_p))
> >> {
> >> printf("error:error reading stat\n");
> >> return 0;
> >> }
> >>
> >> *output_stream_size = stat_p.st_size;
> >> printf("size = %d\n", *output_stream_size);
> >> *output_stream = malloc((*output_stream_size) * sizeof(char));
> >>
> >> if (!(*output_stream))
> >> {
> >> printf("error: cannot create stream\n");
> >> return 0;
> >> }
> >> count = fread(*output_stream, 1, *output_stream_size, f);
> >> if (ferror(f) != 0)
> >> {
> >> printf("error:cannot read file\n");
> >> if (*output_stream)
> >> {
> >> free(*output_stream);
> >> *output_stream = NULL;
> >> }
> >> return 0;
> >>
> >> }
> >> fclose(f);
> >> return 1;
> >> }
> >>
> >>
> >
> >
>
>
Re: [AXIS2] MTOM performance Measuring - http_transport_utils.c modified
Posted by Nabeel Yoosuf <na...@gmail.com>.
Yes, the results are really impressive..if I did the math correctly, in some
cases it is a 30x - 40x gain compared to java!!
-Nabeel
On 5/13/06, Samisa Abeysinghe <sa...@gmail.com> wrote:
>
> Wow!!!
> This is fantastic; At last we are up to the expected speed of C.
>
> Many thanks for looking into this. This has been very useful.
> Please have a look into chunking if you get some time.
> Thanks,
> Samisa...
>
> Kapila Dissanayake wrote:
>
> > Hi ,
> >
> > I did another modification to improve the messege transfer speed.
> >
> > http_transport_utils.c was modified. (Buffer size was changed from
> > 1024 to 1024 * 1024)
> >
> > Now the transfer speed has been changed significantly. Please see the
> > table below.
> >
> >
> > The attached file includes the svn diff of the change.
> >
> > Thank You
> >
> > Kapila
> >
> >
> >
> > (File Size) vs (Time Taken in milli seconds) to send binary attachments
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Environment
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > ----------
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Processor - Intel Celeron M 1.40GHz
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > RAM - 256 MB
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > OS - Fedora Core 4.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Java Server - axis2 Java
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > C Server - axis2
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > mtom C and java test clients were used
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > http_transport_utils.c was modified. (Buffer size was changed from
> > 1024 to 1024 * 1024)
> >
> >
> >
> >
> >
> >
> > Time Taken in milli seconds
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > File Size (kb) Java Server Java
> Server C Server C
> > Server C Server C Server C Server C
> Server C Server C
> > Server
> > Java Client (ms) C Client (ms) C
> Client (ms)
> > Java Client (ms)
> > *data_handler.c Buffer
> Size 1024 * *data_handler.c
> > 'Buffer Size 1024 X 1024 * *Using stat* *http_transport_utils.c
> > 'Buffer Size 1024 X 1024 * * * *Buffer Size '1024 * *Buffer
> Size
> > '1024 X 1024 * *Using stat* *http_transport_utils.c 'Buffer
> Size
> > 1024 X 1024 *
> > ----------------- ----------------- -----------------
> > ----------------- ----------------- -----------------
> > ----------------- ----------------- -----------------
> > --------------------------------- -----------------
> > ----------------- ----------------- -----------------
> > ---------------------------------
> > 13.8 147 158
> 73 1884 78 76 188 143 136 136
> > 28.2 515 488
> 78 1912 78 78 222 413 364 387
> > 37.8 538 694 131
> 1917 81 81 232 474 377 396
> > 132 571 807 212 2177 102
> 83 263 420 381 432
> > 154 547 672 239 2215 117
> 84 347 389 383 517
> > 165 688 683 397 2380 205
> 85 364 400 393 602
> > 561 1340 1024 995 2537 777
> 100 637 899 775 766
> > 762
> 1766 1336 1903 2890 1458 111
> 1248 1218 1051
> > 1055
> >
> 1162 3464 1970 4859 3602 3405 135
> 1702 1718
> > 1712 1092
> > 1761 4666 3274 10928
> 8028 7927 188 2780 3249
> > 2691 2596
> > 2081 6777 2993 15066 11008 10864
> 219 3595 3281
> > 2859 3185
> > 4241 23038 Could not transfer (Java client
> Broken) 52853
> > 33351
> 32957 519 16234 115777
> > 7991 6499
> >
> >
> >
> >
> > On 5/12/06, *Samisa Abeysinghe* <samisa.abeysinghe@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > BTW,
> > If we are going to use chunking at some point in time, then the
> > data
> > handler logic for reading file too has to change to keep track of
> the
> > amount of chunks already read, and current position of read pointer.
> > There is no provision for such reading in the code as of now,
> > neither in
> > the data handler nor in the mime_output that uses the data handler
> > API.
> > So there is considerable amount of work on the part of the data
> hander
> > and the places that uses the API of data handler to read to turn on
> > chunking as far as I can see.
> > Thanks,
> > Samisa...
> >
> > Samisa Abeysinghe wrote:
> >
> > > Nabeel wrote:
> > >
> > >> Kapila,
> > >> With your latest changes you always read the file in one go
> > >> (i.e. not in chunks) and you don't need a loop. The attached
> > compact
> > >> code does the same thing.
> > >>
> > >> Suggestion:
> > >> When we need to attach *very* large file, we may not be keep
> the
> > >> whole file in memory. IMO, this where chunk reading is really
> > useful.
> > >> If you are to implement this, you'll have to change the code from
> > >> data_handler -> mime_body_part -> mime_output to om_output so
> that
> > >> you'll keep only chunk_size amount of bytes in memory at any
> > given time.
> > >
> > >
> > > Chunking is a very good idea, in fact that is the perfect
> > solution for
> > > large attachments.
> > > However, AFAIK, we have chunking capability only on the client
> side
> > > transport, not on the server side transport.
> > > Hence, we got to implement chunking capability into simple axis
> > server
> > > transport first. (May be we can raise a Jira on this). Or we have
> to
> > > see if we can piggyback on Apache for chunking.
> > > Secondly, we got to research and find out the ideal chunk size.
> > As we
> > > have seen through the results sent by Kapila, too small a chunk
> > size
> > > makes too large file transmission very slow and too large chunk
> size
> > > makes small file transmission very slow. Best is to not use
> chunking
> > > for small files and use chunking, with a reasonable chunk size,
> for
> > > large files. So we have several parameters to figure out.
> > > i. Max size we can read a file in one go without chunking.
> > > ii. When chunking is turned on, ideal chunk size.
> > > As I mentioned earlier, we need some research around this to
> > decide on
> > > the figures.
> > >
> > > Thanks,
> > > Samisa...
> > >
> > >>
> > >> -Nabeel
> > >>
> > >> Samisa Abeysinghe wrote:
> > >>
> > >>> Cool, so there sure is an improvement when using stat on read
> > side
> > >>> of data handler. Hence lets use this solution.
> > >>> Still we need to figure out what goes wrong with large files
> > beyond 1M
> > >>>
> > >>> Samisa...
> > >>>
> > >>> Kapila Dissanayake wrote:
> > >>>
> > >>>> Hi
> > >>>>
> > >>>> I did some modification to the data handler. Insted of using
> > fixed
> > >>>> buffer size to read files
> > >>>> 'stat' is used.
> > >>>>
> > >>>> Please see the perfomance details.
> > >>>>
> > >>>> Thank You
> > >>>>
> > >>>> Kapila
> > >>>>
> > >>>>
> > >>>> Environment ---------- Processor -
> > Intel
> > >>>> Celeron M 1.40GHz
> > >>>> RAM - 256 MB OS - Fedora Core 4. Java Server
> -
> > >>>> axis2 Java C Server - axis2 mtom C and java
> > test
> > >>>> clients were used (in axis2 c data_handler.c, different
> > buffer
> > >>>> sizes were used to read files)
> > >>>> Time Taken in milli seconds
> > >>>> File Size (kb) Java Server (ms) Java Server
> > >>>> (ms) C Server (ms) C Server (ms)
> > >>>> Java Client (ms) C Client (ms) C
> > >>>> Client (ms) Java Client (ms)
> > >>>> *Buffer Size 1024 *
> > *Buffer
> > >>>> Size 1024 X 1024 * *Using stat* * * *Buffer Size '1024
> > >>>> * *Buffer Size '1024 X 1024 * *Using stat *
> > >>>> ----------------- ----------------- -----------------
> > >>>> ----------------- ----------------- -----------------
> > >>>> ----------------- ----------------- -----------------
> > >>>> ----------------- ----------------- -----------------
> > >>>> -----------------
> > >>>> 13.8 147 158 73 1884
> > >>>> 78 188 143 136
> > >>>> 28.2 515 488 78 1912
> > >>>> 78 222 413 364
> > >>>> 37.8 538 694 131 1917
> > >>>> 81 232 474 377
> > >>>> 132 571 807 212 2177
> > >>>> 102 263 420 381
> > >>>> 154 547 672 239 2215
> > >>>> 117 347 389 383
> > >>>> 165 688 683 397 2380
> > >>>> 205 364 400 393
> > >>>> 561 1340 1024 995 2537
> > >>>> 777 637 899 775
> > >>>> 762 1766 1336 1903 2890
> > >>>> 1458 1248 1218 1051
> > >>>> 1162 3464 1970 4859 3602
> > >>>> 3405 1702 1718 1712
> > >>>> 1761 4666 3274 10928 8028
> > >>>> 7927 2780 3249 2691
> > >>>> 2081 6777 2993 15066 11008
> > >>>> 10864 3595 3281 2859
> > >>>> 4241 23038 Could not transfer (Java client
> > >>>> Broken) 52853 33351 32957 16234
> > >>>> 115777 7991
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On 5/10/06, *Samisa Abeysinghe* < samisa.abeysinghe@gmail.com
> > <ma...@gmail.com>
> > >>>> <mailto:samisa.abeysinghe@gmail.com
> > <ma...@gmail.com>>> wrote:
> > >>>>
> > >>>> Well, from this I am certain that we should not change the
> > >>>> buffer size
> > >>>> to 1024*1024, rather keep it at 1024.
> > >>>> Please try with stat, using a dynamic buffer size based
> > on the
> > >>>> file
> > >>>> size, my gut feel is that it is going to be the best
> > solution.
> > >>>>
> > >>>> Thanks,
> > >>>> Samisa...
> > >>>>
> > >>>> Kapila Dissanayake wrote:
> > >>>>
> > >>>> >
> > >>>> > Hi,
> > >>>> >
> > >>>> > I did some testing agian with to measure the binary file
> > >>>> transfer
> > >>>> > times. In axis2 c data_handler.c, different buffer sizes
> > >>>> (1024 and
> > >>>> > 1024*1024) were used to read files.
> > >>>> >
> > >>>> > I think using fixed size buffers to read binary files
> > in axis2-C
> > >>>> has
> > >>>> > been affected in some cases to slow down the transfer.
> > >>>> >
> > >>>> > Thanks
> > >>>> >
> > >>>> > Kapila
> > >>>> >
> > >>>> > Please see the following test results
> > >>>> >
> > >>>> > Environment
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > ----------
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > Processor - Intel Celeron M 1.40GHz
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > RAM - 256 MB
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > OS - Fedora Core 4.
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > Java Server - axis2 Java
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > C Server - axis2
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > mtom C and java test clients were used
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > Time Taken in milli seconds
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > File Size (kb) Java Server
> (ms) Java
> > >>>> Server (ms) C
> > >>>> > Server (ms) C Server (ms)
> > >>>> > Java Client (ms) C
> > >>>> Client (ms) C Client (ms)
> > >>>> Java
> > >>>> > Client (ms)
> > >>>> > *Buffer
> Size*
> > >>>> *1024 * *1024 X 1024* * * *1024*
> > >>>> > *1024 X 1024 *
> > >>>> > ----------------- -----------------
> > -----------------
> > >>>> > ----------------- -----------------
> > -----------------
> > >>>> > ----------------- -----------------
> > -----------------
> > >>>> > ----------------- -----------------
> > -----------------
> > >>>> > 13.8 147 158
> > >>>> 73 1884 188 143
> > >>>> > 28.2 515 488
> > >>>> 78 1912 222 413
> > >>>> > 37.8 538 694 131
> > >>>> 1917 232 474
> > >>>> > 132 571 807 212
> > >>>> 2177 263 420
> > >>>> > 154 547 672 239
> > >>>> 2215 347 389
> > >>>> > 165 688 683 397
> > >>>> 2380 364 400
> > >>>> > 561 1340 1024 995
> > >>>> 2537 637 899
> > >>>> > 762 1766 1336
> > >>>> 1903 2890 1248 1218
> > >>>> >
> > >>>> 1162 3464 1970 4859
> > >>>> 3602 1702 1718
> > >>>>
> > >>>> >
> 1761 4666 3274 10928
> > >>>> 8028 2780 3249
> > >>>> >
> > 2081 6777 2993 15066
> > >>>> 11008 3595 3281
> > >>>> > 4241 23038 Could not transfer (Java
> > client
> > >>>> Broken)
> > >>>> > 52853 33351 16234 115777
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > On 5/8/06, *Samisa Abeysinghe* <samisa@wso2.com
> > <ma...@wso2.com>
> > >>>> <mailto:samisa@wso2.com <ma...@wso2.com>>
> > >>>> > <mailto: samisa@wso2.com <ma...@wso2.com>
> > <mailto: samisa@wso2.com <ma...@wso2.com>>>> wrote:
> > >>>> >
> > >>>> > Hmmm, Interesting. There seem to be a problem in C
> > client in
> > >>>> case of
> > >>>> > large attachments.
> > >>>> > This must be due to the fact that we are using a
> > loop to
> > >>>> read the
> > >>>> > file.
> > >>>> > If we stat and read the file at once, there could
> > be an
> > >>>> improvement.
> > >>>> >
> > >>>> > Why is it so slow, in C vs C case? (Third column)
> > This is
> > >>>> worth
> > >>>> > investigating.
> > >>>> >
> > >>>> > Thanks,
> > >>>> > Samisa...
> > >>>> >
> > >>>> > Kapila Dissanayake wrote:
> > >>>> >
> > >>>> > > Hi,
> > >>>> > >
> > >>>> > > I did some testing on binary attachment transfer
> > times,
> > >>>> with axis2
> > >>>> > > 'java' and 'C' servers with sample mtom clients.
> > It took
> > >>>> very high
> > >>>> > > transfer times for large size attachments. Some
> minor
> > >>>> modificaions
> > >>>> > > were done to the data_handler.c and it was sent
> > to the
> > >>>> mailing list.
> > >>>> > >
> > >>>> > >
> > >>>> > > (File Size) vs (Time Taken in milli seconds) to
> > transfer
> > >>>> binary
> > >>>> > > attachments
> > >>>> > >
> > >>>> > > Environment
> > >>>> > > ----------
> > >>>> > > Processor - Intel Celeron M 1.40GHz
> > >>>> > > RAM - 256 MB
> > >>>> > > OS - Fedora Core 4.
> > >>>> > > Java Server - axis2 Java
> > >>>> > > C Server - axis2
> > >>>> > > mtom C and java test clients were used
> > >>>> > > Time Taken in milli seconds
> > >>>> > >
> > >>>> > > File Size (kb)
> > >>>> > >
> > >>>> > > Java Server
> > >>>> > >
> > >>>> > > (ms)
> > >>>> > >
> > >>>> > > Java Server
> > >>>> > >
> > >>>> > > (ms)
> > >>>> > >
> > >>>> > > C Server
> > >>>> > >
> > >>>> > > (ms)
> > >>>> > >
> > >>>> > > C Server
> > >>>> > >
> > >>>> > > (ms)
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > Java Client
> > >>>> > >
> > >>>> > > (ms)
> > >>>> > >
> > >>>> > > C Client
> > >>>> > >
> > >>>> > > (ms)
> > >>>> > >
> > >>>> > > C Client
> > >>>> > >
> > >>>> > > (ms)
> > >>>> > >
> > >>>> > > Java Client
> > >>>> > >
> > >>>> > > (ms)
> > >>>> > > ---------------
> > >>>> > >
> > >>>> > > ---------------
> > >>>> > >
> > >>>> > > ---------------
> > >>>> > >
> > >>>> > > ---------------
> > >>>> > >
> > >>>> > > ---------------
> > >>>> > >
> > >>>> > > ---------------
> > >>>> > >
> > >>>> > > ---------------
> > >>>> > >
> > >>>> > > ---------------
> > >>>> > >
> > >>>> > > ---------------
> > >>>> > > 28
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 445
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 274
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 2004
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 1009
> > >>>> > >
> > >>>> > >
> > >>>> > > 38
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 617
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 297
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 1957
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 842
> > >>>> > >
> > >>>> > >
> > >>>> > > 132
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 593
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 527
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 2181
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 726
> > >>>> > >
> > >>>> > >
> > >>>> > > 154
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 551
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 447
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 2081
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 677
> > >>>> > >
> > >>>> > >
> > >>>> > > 165
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 440
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 743
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 2147
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 676
> > >>>> > >
> > >>>> > >
> > >>>> > > 561
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 1029
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 418
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 2581
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 1075
> > >>>> > >
> > >>>> > >
> > >>>> > > 762
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 2137
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 1572
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 3231
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 2356
> > >>>> > >
> > >>>> > >
> > >>>> > > 1162
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 3172
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 2962
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 3981
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 3100
> > >>>> > >
> > >>>> > >
> > >>>> > > 1761
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 7672
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 5494
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 8186
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 3041
> > >>>> > >
> > >>>> > >
> > >>>> > > 2081
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 7768
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 6837
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 11757
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 3499
> > >>>> > >
> > >>>> > >
> > >>>> > > 4241
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 14465
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 24060
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > 36168
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > Could not transfer (Error Occured in Java client)
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > Thank You
> > >>>> > >
> > >>>> > > Kapila
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>>
> > >>>>
> > >>>
> > >>>
> > >>
> > >>
> >
> ------------------------------------------------------------------------
> >
> > >>
> > >> #include <stdio.h>
> > >> #include <sys/stat.h>
> > >>
> > >> int read_from_file(char *file_name, char** output_stream, int
> > >> *output_stream_size);
> > >>
> > >> int main(int argc, char **argv)
> > >> {
> > >> int size;
> > >> char *output;
> > >>
> > >> if (argc != 2)
> > >> {
> > >> printf("usage: read_binary <file_name>\n");
> > >> return 0;
> > >> }
> > >> if (read_from_file(argv[1], &output, &size))
> > >> printf("read successful\n");
> > >> if (output)
> > >> free(output);
> > >> return 1;
> > >> }
> > >>
> > >> int read_from_file(char *file_name, char** output_stream, int
> > >> *output_stream_size)
> > >> {
> > >> FILE *f = NULL;
> > >> struct stat stat_p;
> > >> int count = 0;
> > >>
> > >> if (!file_name)
> > >> {
> > >> printf("error: file not given \n");
> > >> return 0;
> > >> }
> > >> f = fopen(file_name, "rb");
> > >> if (!f)
> > >> {
> > >> printf("error:cannot open file %s\n", file_name);
> > >> return ;
> > >> }
> > >> if ( -1 == stat (file_name, &stat_p))
> > >> {
> > >> printf("error:error reading stat\n");
> > >> return 0;
> > >> }
> > >>
> > >> *output_stream_size = stat_p.st_size;
> > >> printf("size = %d\n", *output_stream_size);
> > >> *output_stream = malloc((*output_stream_size) *
> sizeof(char));
> > >>
> > >> if (!(*output_stream))
> > >> {
> > >> printf("error: cannot create stream\n");
> > >> return 0;
> > >> }
> > >> count = fread(*output_stream, 1, *output_stream_size, f);
> > >> if (ferror(f) != 0)
> > >> {
> > >> printf("error:cannot read file\n");
> > >> if (*output_stream)
> > >> {
> > >> free(*output_stream);
> > >> *output_stream = NULL;
> > >> }
> > >> return 0;
> > >>
> > >> }
> > >> fclose(f);
> > >> return 1;
> > >> }
> > >>
> > >>
> > >
> > >
> >
> >
> >------------------------------------------------------------------------
> >
> >Index: modules/xml/attachments/data_handler.c
> >===================================================================
> >--- modules/xml/attachments/data_handler.c (revision 398104)
> >+++ modules/xml/attachments/data_handler.c (working copy)
> >@@ -16,6 +16,7 @@
> >
> > #include <axis2_data_handler.h>
> > #include <stdio.h>
> >+#include <sys/stat.h>
> >
> > typedef struct axis2_data_handler_impl
> > {
> >@@ -185,7 +186,12 @@
> > return AXIS2_FAILURE;
> >
> > do {
> >- read_stream_size = 1024;
> >+ struct stat stat_p;
> >+ if ( -1 == stat (data_handler_impl->file_name, &stat_p))
> >+ {
> >+ return AXIS2_FAILURE;
> >+ }
> >+ read_stream_size = stat_p.st_size;
> > read_stream = AXIS2_MALLOC((*env)->allocator,
> (read_stream_size) * sizeof(axis2_byte_t));
> > if (!read_stream)
> > {
> >@@ -197,7 +203,7 @@
> > }
> > return AXIS2_FAILURE;
> > }
> >- count = fread(read_stream, 1, 1024, f);
> >+ count = fread(read_stream, 1, read_stream_size, f);
> > if (ferror(f) != 0)
> > {
> > /*TODO : need to set the correct error code */
> >@@ -354,3 +360,4 @@
> > return AXIS2_SUCCESS;
> > }
> >
> >+
> >Index: modules/core/transport/http/http_transport_utils.c
> >===================================================================
> >--- modules/core/transport/http/http_transport_utils.c (revision 398104)
> >+++ modules/core/transport/http/http_transport_utils.c (working copy)
> >@@ -199,7 +199,7 @@
> > if (mime_boundary)
> > {
> > axis2_char_t *buffer = NULL;
> >- int size = 1024;
> >+ int size = 1024 * 1024;
> > int len = 0;
> > axis2_char_t *root_mime = NULL;
> > int root_mime_len = 0;
> >Index: build.sh
> >===================================================================
> >--- build.sh (revision 398104)
> >+++ build.sh (working copy)
> >@@ -2,7 +2,7 @@
> > ./autogen.sh
> > AXIS2C_HOME=`pwd`/deploy
> > export AXIS2C_HOME
> >-./configure --prefix=${AXIS2C_HOME} --enable-tests=yes
> --enable-trace=yes --with-apache2=/usr/include/apache2
> --with-apr=/usr/include/apr-0 --enable-diclient=no --enable-static=no
> >+./configure --prefix=${AXIS2C_HOME} --enable-tests=yes
> --enable-trace=yes --enable-diclient=no --enable-static=no
> > make
> > make install
> > cd samples
> >
> >
>
>
Re: [AXIS2] MTOM performance Measuring - http_transport_utils.c modified
Posted by Samisa Abeysinghe <sa...@gmail.com>.
Wow!!!
This is fantastic; At last we are up to the expected speed of C.
Many thanks for looking into this. This has been very useful.
Please have a look into chunking if you get some time.
Thanks,
Samisa...
Kapila Dissanayake wrote:
> Hi ,
>
> I did another modification to improve the messege transfer speed.
>
> http_transport_utils.c was modified. (Buffer size was changed from
> 1024 to 1024 * 1024)
>
> Now the transfer speed has been changed significantly. Please see the
> table below.
>
>
> The attached file includes the svn diff of the change.
>
> Thank You
>
> Kapila
>
>
>
> (File Size) vs (Time Taken in milli seconds) to send binary attachments
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Environment
>
>
>
>
>
>
>
>
>
>
>
>
> ----------
>
>
>
>
>
>
>
>
>
>
>
>
>
> Processor - Intel Celeron M 1.40GHz
>
>
>
>
>
>
>
>
>
>
> RAM - 256 MB
>
>
>
>
>
>
>
>
>
>
>
>
> OS - Fedora Core 4.
>
>
>
>
>
>
>
>
>
>
>
> Java Server - axis2 Java
>
>
>
>
>
>
>
>
>
>
>
> C Server - axis2
>
>
>
>
>
>
>
>
>
>
>
>
> mtom C and java test clients were used
>
>
>
>
>
>
>
>
>
>
> http_transport_utils.c was modified. (Buffer size was changed from
> 1024 to 1024 * 1024)
>
>
>
>
>
>
> Time Taken in milli seconds
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> File Size (kb) Java Server Java Server C Server C
> Server C Server C Server C Server C Server C Server C
> Server
> Java Client (ms) C Client (ms) C Client (ms)
> Java Client (ms)
> *data_handler.c Buffer Size 1024 * *data_handler.c
> 'Buffer Size 1024 X 1024 * *Using stat* *http_transport_utils.c
> 'Buffer Size 1024 X 1024 * * * *Buffer Size '1024 * *Buffer Size
> '1024 X 1024 * *Using stat* *http_transport_utils.c 'Buffer Size
> 1024 X 1024 *
> ----------------- ----------------- -----------------
> ----------------- ----------------- -----------------
> ----------------- ----------------- -----------------
> --------------------------------- -----------------
> ----------------- ----------------- -----------------
> ---------------------------------
> 13.8 147 158 73 1884 78 76 188 143 136 136
> 28.2 515 488 78 1912 78 78 222 413 364 387
> 37.8 538 694 131 1917 81 81 232 474 377 396
> 132 571 807 212 2177 102 83 263 420 381 432
> 154 547 672 239 2215 117 84 347 389 383 517
> 165 688 683 397 2380 205 85 364 400 393 602
> 561 1340 1024 995 2537 777 100 637 899 775 766
> 762 1766 1336 1903 2890 1458 111 1248 1218 1051
> 1055
> 1162 3464 1970 4859 3602 3405 135 1702 1718
> 1712 1092
> 1761 4666 3274 10928 8028 7927 188 2780 3249
> 2691 2596
> 2081 6777 2993 15066 11008 10864 219 3595 3281
> 2859 3185
> 4241 23038 Could not transfer (Java client Broken) 52853
> 33351 32957 519 16234 115777
> 7991 6499
>
>
>
>
> On 5/12/06, *Samisa Abeysinghe* <samisa.abeysinghe@gmail.com
> <ma...@gmail.com>> wrote:
>
> BTW,
> If we are going to use chunking at some point in time, then the
> data
> handler logic for reading file too has to change to keep track of the
> amount of chunks already read, and current position of read pointer.
> There is no provision for such reading in the code as of now,
> neither in
> the data handler nor in the mime_output that uses the data handler
> API.
> So there is considerable amount of work on the part of the data hander
> and the places that uses the API of data handler to read to turn on
> chunking as far as I can see.
> Thanks,
> Samisa...
>
> Samisa Abeysinghe wrote:
>
> > Nabeel wrote:
> >
> >> Kapila,
> >> With your latest changes you always read the file in one go
> >> (i.e. not in chunks) and you don't need a loop. The attached
> compact
> >> code does the same thing.
> >>
> >> Suggestion:
> >> When we need to attach *very* large file, we may not be keep the
> >> whole file in memory. IMO, this where chunk reading is really
> useful.
> >> If you are to implement this, you'll have to change the code from
> >> data_handler -> mime_body_part -> mime_output to om_output so that
> >> you'll keep only chunk_size amount of bytes in memory at any
> given time.
> >
> >
> > Chunking is a very good idea, in fact that is the perfect
> solution for
> > large attachments.
> > However, AFAIK, we have chunking capability only on the client side
> > transport, not on the server side transport.
> > Hence, we got to implement chunking capability into simple axis
> server
> > transport first. (May be we can raise a Jira on this). Or we have to
> > see if we can piggyback on Apache for chunking.
> > Secondly, we got to research and find out the ideal chunk size.
> As we
> > have seen through the results sent by Kapila, too small a chunk
> size
> > makes too large file transmission very slow and too large chunk size
> > makes small file transmission very slow. Best is to not use chunking
> > for small files and use chunking, with a reasonable chunk size, for
> > large files. So we have several parameters to figure out.
> > i. Max size we can read a file in one go without chunking.
> > ii. When chunking is turned on, ideal chunk size.
> > As I mentioned earlier, we need some research around this to
> decide on
> > the figures.
> >
> > Thanks,
> > Samisa...
> >
> >>
> >> -Nabeel
> >>
> >> Samisa Abeysinghe wrote:
> >>
> >>> Cool, so there sure is an improvement when using stat on read
> side
> >>> of data handler. Hence lets use this solution.
> >>> Still we need to figure out what goes wrong with large files
> beyond 1M
> >>>
> >>> Samisa...
> >>>
> >>> Kapila Dissanayake wrote:
> >>>
> >>>> Hi
> >>>>
> >>>> I did some modification to the data handler. Insted of using
> fixed
> >>>> buffer size to read files
> >>>> 'stat' is used.
> >>>>
> >>>> Please see the perfomance details.
> >>>>
> >>>> Thank You
> >>>>
> >>>> Kapila
> >>>>
> >>>>
> >>>> Environment ---------- Processor -
> Intel
> >>>> Celeron M 1.40GHz
> >>>> RAM - 256 MB OS - Fedora Core 4. Java Server -
> >>>> axis2 Java C Server - axis2 mtom C and java
> test
> >>>> clients were used (in axis2 c data_handler.c, different
> buffer
> >>>> sizes were used to read files)
> >>>> Time Taken in milli seconds
> >>>> File Size (kb) Java Server (ms) Java Server
> >>>> (ms) C Server (ms) C Server (ms)
> >>>> Java Client (ms) C Client (ms) C
> >>>> Client (ms) Java Client (ms)
> >>>> *Buffer Size 1024 *
> *Buffer
> >>>> Size 1024 X 1024 * *Using stat* * * *Buffer Size '1024
> >>>> * *Buffer Size '1024 X 1024 * *Using stat *
> >>>> ----------------- ----------------- -----------------
> >>>> ----------------- ----------------- -----------------
> >>>> ----------------- ----------------- -----------------
> >>>> ----------------- ----------------- -----------------
> >>>> -----------------
> >>>> 13.8 147 158 73 1884
> >>>> 78 188 143 136
> >>>> 28.2 515 488 78 1912
> >>>> 78 222 413 364
> >>>> 37.8 538 694 131 1917
> >>>> 81 232 474 377
> >>>> 132 571 807 212 2177
> >>>> 102 263 420 381
> >>>> 154 547 672 239 2215
> >>>> 117 347 389 383
> >>>> 165 688 683 397 2380
> >>>> 205 364 400 393
> >>>> 561 1340 1024 995 2537
> >>>> 777 637 899 775
> >>>> 762 1766 1336 1903 2890
> >>>> 1458 1248 1218 1051
> >>>> 1162 3464 1970 4859 3602
> >>>> 3405 1702 1718 1712
> >>>> 1761 4666 3274 10928 8028
> >>>> 7927 2780 3249 2691
> >>>> 2081 6777 2993 15066 11008
> >>>> 10864 3595 3281 2859
> >>>> 4241 23038 Could not transfer (Java client
> >>>> Broken) 52853 33351 32957 16234
> >>>> 115777 7991
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 5/10/06, *Samisa Abeysinghe* < samisa.abeysinghe@gmail.com
> <ma...@gmail.com>
> >>>> <mailto:samisa.abeysinghe@gmail.com
> <ma...@gmail.com>>> wrote:
> >>>>
> >>>> Well, from this I am certain that we should not change the
> >>>> buffer size
> >>>> to 1024*1024, rather keep it at 1024.
> >>>> Please try with stat, using a dynamic buffer size based
> on the
> >>>> file
> >>>> size, my gut feel is that it is going to be the best
> solution.
> >>>>
> >>>> Thanks,
> >>>> Samisa...
> >>>>
> >>>> Kapila Dissanayake wrote:
> >>>>
> >>>> >
> >>>> > Hi,
> >>>> >
> >>>> > I did some testing agian with to measure the binary file
> >>>> transfer
> >>>> > times. In axis2 c data_handler.c, different buffer sizes
> >>>> (1024 and
> >>>> > 1024*1024) were used to read files.
> >>>> >
> >>>> > I think using fixed size buffers to read binary files
> in axis2-C
> >>>> has
> >>>> > been affected in some cases to slow down the transfer.
> >>>> >
> >>>> > Thanks
> >>>> >
> >>>> > Kapila
> >>>> >
> >>>> > Please see the following test results
> >>>> >
> >>>> > Environment
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > ----------
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > Processor - Intel Celeron M 1.40GHz
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > RAM - 256 MB
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > OS - Fedora Core 4.
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > Java Server - axis2 Java
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > C Server - axis2
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > mtom C and java test clients were used
> >>>> >
> >>>> >
> >>>> >
> >>>> > Time Taken in milli seconds
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > File Size (kb) Java Server (ms) Java
> >>>> Server (ms) C
> >>>> > Server (ms) C Server (ms)
> >>>> > Java Client (ms) C
> >>>> Client (ms) C Client (ms)
> >>>> Java
> >>>> > Client (ms)
> >>>> > *Buffer Size*
> >>>> *1024 * *1024 X 1024* * * *1024*
> >>>> > *1024 X 1024 *
> >>>> > ----------------- -----------------
> -----------------
> >>>> > ----------------- -----------------
> -----------------
> >>>> > ----------------- -----------------
> -----------------
> >>>> > ----------------- -----------------
> -----------------
> >>>> > 13.8 147 158
> >>>> 73 1884 188 143
> >>>> > 28.2 515 488
> >>>> 78 1912 222 413
> >>>> > 37.8 538 694 131
> >>>> 1917 232 474
> >>>> > 132 571 807 212
> >>>> 2177 263 420
> >>>> > 154 547 672 239
> >>>> 2215 347 389
> >>>> > 165 688 683 397
> >>>> 2380 364 400
> >>>> > 561 1340 1024 995
> >>>> 2537 637 899
> >>>> > 762 1766 1336
> >>>> 1903 2890 1248 1218
> >>>> >
> >>>> 1162 3464 1970 4859
> >>>> 3602 1702 1718
> >>>>
> >>>> > 1761 4666 3274 10928
> >>>> 8028 2780 3249
> >>>> >
> 2081 6777 2993 15066
> >>>> 11008 3595 3281
> >>>> > 4241 23038 Could not transfer (Java
> client
> >>>> Broken)
> >>>> > 52853 33351 16234 115777
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > On 5/8/06, *Samisa Abeysinghe* <samisa@wso2.com
> <ma...@wso2.com>
> >>>> <mailto:samisa@wso2.com <ma...@wso2.com>>
> >>>> > <mailto: samisa@wso2.com <ma...@wso2.com>
> <mailto: samisa@wso2.com <ma...@wso2.com>>>> wrote:
> >>>> >
> >>>> > Hmmm, Interesting. There seem to be a problem in C
> client in
> >>>> case of
> >>>> > large attachments.
> >>>> > This must be due to the fact that we are using a
> loop to
> >>>> read the
> >>>> > file.
> >>>> > If we stat and read the file at once, there could
> be an
> >>>> improvement.
> >>>> >
> >>>> > Why is it so slow, in C vs C case? (Third column)
> This is
> >>>> worth
> >>>> > investigating.
> >>>> >
> >>>> > Thanks,
> >>>> > Samisa...
> >>>> >
> >>>> > Kapila Dissanayake wrote:
> >>>> >
> >>>> > > Hi,
> >>>> > >
> >>>> > > I did some testing on binary attachment transfer
> times,
> >>>> with axis2
> >>>> > > 'java' and 'C' servers with sample mtom clients.
> It took
> >>>> very high
> >>>> > > transfer times for large size attachments. Some minor
> >>>> modificaions
> >>>> > > were done to the data_handler.c and it was sent
> to the
> >>>> mailing list.
> >>>> > >
> >>>> > >
> >>>> > > (File Size) vs (Time Taken in milli seconds) to
> transfer
> >>>> binary
> >>>> > > attachments
> >>>> > >
> >>>> > > Environment
> >>>> > > ----------
> >>>> > > Processor - Intel Celeron M 1.40GHz
> >>>> > > RAM - 256 MB
> >>>> > > OS - Fedora Core 4.
> >>>> > > Java Server - axis2 Java
> >>>> > > C Server - axis2
> >>>> > > mtom C and java test clients were used
> >>>> > > Time Taken in milli seconds
> >>>> > >
> >>>> > > File Size (kb)
> >>>> > >
> >>>> > > Java Server
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > > Java Server
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > > C Server
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > > C Server
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Java Client
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > > C Client
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > > C Client
> >>>> > >
> >>>> > > (ms)
> >>>> > >
> >>>> > > Java Client
> >>>> > >
> >>>> > > (ms)
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > >
> >>>> > > ---------------
> >>>> > > 28
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 445
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 274
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2004
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 1009
> >>>> > >
> >>>> > >
> >>>> > > 38
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 617
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 297
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 1957
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 842
> >>>> > >
> >>>> > >
> >>>> > > 132
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 593
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 527
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2181
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 726
> >>>> > >
> >>>> > >
> >>>> > > 154
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 551
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 447
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2081
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 677
> >>>> > >
> >>>> > >
> >>>> > > 165
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 440
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 743
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2147
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 676
> >>>> > >
> >>>> > >
> >>>> > > 561
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 1029
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 418
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2581
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 1075
> >>>> > >
> >>>> > >
> >>>> > > 762
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2137
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 1572
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 3231
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2356
> >>>> > >
> >>>> > >
> >>>> > > 1162
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 3172
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2962
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 3981
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 3100
> >>>> > >
> >>>> > >
> >>>> > > 1761
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 7672
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 5494
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 8186
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 3041
> >>>> > >
> >>>> > >
> >>>> > > 2081
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 7768
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 6837
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 11757
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 3499
> >>>> > >
> >>>> > >
> >>>> > > 4241
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 14465
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 24060
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 36168
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Could not transfer (Error Occured in Java client)
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Thank You
> >>>> > >
> >>>> > > Kapila
> >>>> >
> >>>> >
> >>>> >
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> ------------------------------------------------------------------------
>
> >>
> >> #include <stdio.h>
> >> #include <sys/stat.h>
> >>
> >> int read_from_file(char *file_name, char** output_stream, int
> >> *output_stream_size);
> >>
> >> int main(int argc, char **argv)
> >> {
> >> int size;
> >> char *output;
> >>
> >> if (argc != 2)
> >> {
> >> printf("usage: read_binary <file_name>\n");
> >> return 0;
> >> }
> >> if (read_from_file(argv[1], &output, &size))
> >> printf("read successful\n");
> >> if (output)
> >> free(output);
> >> return 1;
> >> }
> >>
> >> int read_from_file(char *file_name, char** output_stream, int
> >> *output_stream_size)
> >> {
> >> FILE *f = NULL;
> >> struct stat stat_p;
> >> int count = 0;
> >>
> >> if (!file_name)
> >> {
> >> printf("error: file not given \n");
> >> return 0;
> >> }
> >> f = fopen(file_name, "rb");
> >> if (!f)
> >> {
> >> printf("error:cannot open file %s\n", file_name);
> >> return ;
> >> }
> >> if ( -1 == stat (file_name, &stat_p))
> >> {
> >> printf("error:error reading stat\n");
> >> return 0;
> >> }
> >>
> >> *output_stream_size = stat_p.st_size;
> >> printf("size = %d\n", *output_stream_size);
> >> *output_stream = malloc((*output_stream_size) * sizeof(char));
> >>
> >> if (!(*output_stream))
> >> {
> >> printf("error: cannot create stream\n");
> >> return 0;
> >> }
> >> count = fread(*output_stream, 1, *output_stream_size, f);
> >> if (ferror(f) != 0)
> >> {
> >> printf("error:cannot read file\n");
> >> if (*output_stream)
> >> {
> >> free(*output_stream);
> >> *output_stream = NULL;
> >> }
> >> return 0;
> >>
> >> }
> >> fclose(f);
> >> return 1;
> >> }
> >>
> >>
> >
> >
>
>
>------------------------------------------------------------------------
>
>Index: modules/xml/attachments/data_handler.c
>===================================================================
>--- modules/xml/attachments/data_handler.c (revision 398104)
>+++ modules/xml/attachments/data_handler.c (working copy)
>@@ -16,6 +16,7 @@
>
> #include <axis2_data_handler.h>
> #include <stdio.h>
>+#include <sys/stat.h>
>
> typedef struct axis2_data_handler_impl
> {
>@@ -185,7 +186,12 @@
> return AXIS2_FAILURE;
>
> do {
>- read_stream_size = 1024;
>+ struct stat stat_p;
>+ if ( -1 == stat (data_handler_impl->file_name, &stat_p))
>+ {
>+ return AXIS2_FAILURE;
>+ }
>+ read_stream_size = stat_p.st_size;
> read_stream = AXIS2_MALLOC((*env)->allocator, (read_stream_size) * sizeof(axis2_byte_t));
> if (!read_stream)
> {
>@@ -197,7 +203,7 @@
> }
> return AXIS2_FAILURE;
> }
>- count = fread(read_stream, 1, 1024, f);
>+ count = fread(read_stream, 1, read_stream_size, f);
> if (ferror(f) != 0)
> {
> /*TODO : need to set the correct error code */
>@@ -354,3 +360,4 @@
> return AXIS2_SUCCESS;
> }
>
>+
>Index: modules/core/transport/http/http_transport_utils.c
>===================================================================
>--- modules/core/transport/http/http_transport_utils.c (revision 398104)
>+++ modules/core/transport/http/http_transport_utils.c (working copy)
>@@ -199,7 +199,7 @@
> if (mime_boundary)
> {
> axis2_char_t *buffer = NULL;
>- int size = 1024;
>+ int size = 1024 * 1024;
> int len = 0;
> axis2_char_t *root_mime = NULL;
> int root_mime_len = 0;
>Index: build.sh
>===================================================================
>--- build.sh (revision 398104)
>+++ build.sh (working copy)
>@@ -2,7 +2,7 @@
> ./autogen.sh
> AXIS2C_HOME=`pwd`/deploy
> export AXIS2C_HOME
>-./configure --prefix=${AXIS2C_HOME} --enable-tests=yes --enable-trace=yes --with-apache2=/usr/include/apache2 --with-apr=/usr/include/apr-0 --enable-diclient=no --enable-static=no
>+./configure --prefix=${AXIS2C_HOME} --enable-tests=yes --enable-trace=yes --enable-diclient=no --enable-static=no
> make
> make install
> cd samples
>
>
Re: [AXIS2] MTOM performance Measuring - http_transport_utils.c modified
Posted by Samisa Abeysinghe <sa...@gmail.com>.
Thilina Gunarathne wrote:
> Wow Wow... pretty impressive.. Way to go guys... Keep up the good work..
>
> It'll be great if we can give it a try to benchmark our stuff against
> WCF ,Xfire , etc too...
>
> Have you already implemented the Base64 conversion to happen
> automatically when the binary is not optimized.. It's critical to have
> it before we claim about the XOP/MTOM feature.
Base64 is a TODO. There are some other issues reported in Jira as well.
Samisa...
>
> ~Thilina
>
> On 5/13/06, *Kapila Dissanayake* <kapidis@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi ,
>
> I did another modification to improve the messege transfer speed.
>
> http_transport_utils.c was modified. (Buffer size was changed from
> 1024 to 1024 * 1024)
>
> Now the transfer speed has been changed significantly. Please see
> the table below.
>
>
> The attached file includes the svn diff of the change.
>
> Thank You
>
> Kapila
>
>
>
> (File Size) vs (Time Taken in milli seconds) to send binary
> attachments
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Environment
>
>
>
>
>
>
>
>
>
>
>
>
> ----------
>
>
>
>
>
>
>
>
>
>
>
>
>
> Processor - Intel Celeron M 1.40GHz
>
>
>
>
>
>
>
>
>
>
> RAM - 256 MB
>
>
>
>
>
>
>
>
>
>
>
>
> OS - Fedora Core 4.
>
>
>
>
>
>
>
>
>
>
>
> Java Server - axis2 Java
>
>
>
>
>
>
>
>
>
>
>
> C Server - axis2
>
>
>
>
>
>
>
>
>
>
>
>
> mtom C and java test clients were used
>
>
>
>
>
>
>
>
>
>
> http_transport_utils.c was modified. (Buffer size was changed from
> 1024 to 1024 * 1024)
>
>
>
>
>
>
> Time Taken in milli seconds
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> File Size (kb) Java Server Java Server C Server C
> Server C Server C Server C Server C Server C Server
> C Server
> Java Client (ms) C Client (ms) C Client
> (ms) Java Client (ms)
> *data_handler.c Buffer Size 1024 *
> *data_handler.c 'Buffer Size 1024 X 1024 * *Using stat*
> *http_transport_utils.c 'Buffer Size 1024 X 1024 * * * *Buffer
> Size '1024 * *Buffer Size '1024 X 1024 * *Using stat*
> *http_transport_utils.c 'Buffer Size 1024 X 1024 *
> ----------------- ----------------- -----------------
> ----------------- ----------------- -----------------
> ----------------- ----------------- -----------------
> --------------------------------- -----------------
> ----------------- ----------------- -----------------
> ---------------------------------
> 13.8 147 158 73 1884 78 76 188 143 136 136
> 28.2 515 488 78 1912 78 78 222 413 364 387
> 37.8 538 694 131 1917 81 81 232 474 377 396
> 132 571 807 212 2177 102 83 263 420 381 432
> 154 547 672 239 2215 117 84 347 389 383 517
> 165 688 683 397 2380 205 85 364 400 393 602
> 561 1340 1024 995 2537 777 100 637 899 775 766
> 762 1766 1336 1903 2890 1458 111 1248 1218
> 1051 1055
> 1162 3464 1970 4859 3602 3405 135 1702
> 1718 1712 1092
> 1761 4666 3274 10928 8028 7927 188 2780 3249
> 2691 2596
> 2081 6777 2993 15066 11008 10864 219 3595
> 3281 2859 3185
> 4241 23038 Could not transfer (Java client Broken)
> 52853 33351 32957 519 16234
> 115777 7991 6499
>
>
>
>
> On 5/12/06, *Samisa Abeysinghe* <samisa.abeysinghe@gmail.com
> <ma...@gmail.com>> wrote:
>
> BTW,
> If we are going to use chunking at some point in time, then
> the data
> handler logic for reading file too has to change to keep track
> of the
> amount of chunks already read, and current position of read
> pointer.
> There is no provision for such reading in the code as of now,
> neither in
> the data handler nor in the mime_output that uses the data
> handler API.
> So there is considerable amount of work on the part of the
> data hander
> and the places that uses the API of data handler to read to
> turn on
> chunking as far as I can see.
> Thanks,
> Samisa...
>
> Samisa Abeysinghe wrote:
>
> > Nabeel wrote:
> >
> >> Kapila,
> >> With your latest changes you always read the file in one go
> >> (i.e. not in chunks) and you don't need a loop. The
> attached compact
> >> code does the same thing.
> >>
> >> Suggestion:
> >> When we need to attach *very* large file, we may not be
> keep the
> >> whole file in memory. IMO, this where chunk reading is
> really useful.
> >> If you are to implement this, you'll have to change the
> code from
> >> data_handler -> mime_body_part -> mime_output to om_output
> so that
> >> you'll keep only chunk_size amount of bytes in memory at
> any given time.
> >
> >
> > Chunking is a very good idea, in fact that is the perfect
> solution for
> > large attachments.
> > However, AFAIK, we have chunking capability only on the
> client side
> > transport, not on the server side transport.
> > Hence, we got to implement chunking capability into simple
> axis server
> > transport first. (May be we can raise a Jira on this). Or we
> have to
> > see if we can piggyback on Apache for chunking.
> > Secondly, we got to research and find out the ideal chunk
> size. As we
> > have seen through the results sent by Kapila, too small a
> chunk size
> > makes too large file transmission very slow and too large
> chunk size
> > makes small file transmission very slow. Best is to not use
> chunking
> > for small files and use chunking, with a reasonable chunk
> size, for
> > large files. So we have several parameters to figure out.
> > i. Max size we can read a file in one go without chunking.
> > ii. When chunking is turned on, ideal chunk size.
> > As I mentioned earlier, we need some research around this to
> decide on
> > the figures.
> >
> > Thanks,
> > Samisa...
> >
> >>
> >> -Nabeel
> >>
> >> Samisa Abeysinghe wrote:
> >>
> >>> Cool, so there sure is an improvement when using stat on
> read side
> >>> of data handler. Hence lets use this solution.
> >>> Still we need to figure out what goes wrong with large
> files beyond 1M
> >>>
> >>> Samisa...
> >>>
> >>> Kapila Dissanayake wrote:
> >>>
> >>>> Hi
> >>>>
> >>>> I did some modification to the data handler. Insted of
> using fixed
> >>>> buffer size to read files
> >>>> 'stat' is used.
> >>>>
> >>>> Please see the perfomance details.
> >>>>
> >>>> Thank You
> >>>>
> >>>> Kapila
> >>>>
> >>>>
> >>>>
> Environment ---------- Processor -
> Intel
> >>>> Celeron M 1.40GHz
> >>>> RAM - 256 MB OS - Fedora Core 4. Java
> Server -
> >>>> axis2 Java C Server - axis2 mtom C and
> java test
> >>>> clients were used (in axis2 c data_handler.c,
> different buffer
> >>>> sizes were used to read files)
> >>>> Time Taken in milli seconds
> >>>> File Size (kb) Java Server (ms) Java Server
> >>>> (ms) C Server (ms) C Server (ms)
> >>>> Java Client (ms) C Client (ms) C
> >>>> Client (ms) Java Client (ms)
> >>>> *Buffer Size 1024
> * *Buffer
> >>>> Size 1024 X 1024 * *Using stat* * * *Buffer Size
> '1024
> >>>> * *Buffer Size '1024 X 1024 * *Using stat *
> >>>> ----------------- -----------------
> -----------------
> >>>> ----------------- ----------------- -----------------
> >>>> ----------------- ----------------- -----------------
> >>>> ----------------- -----------------
> -----------------
> >>>> -----------------
> >>>> 13.8 147 158 73 1884
> >>>> 78 188 143 136
> >>>> 28.2 515 488 78 1912
> >>>> 78 222 413 364
> >>>> 37.8 538 694 131 1917
> >>>> 81 232 474 377
> >>>> 132 571 807 212 2177
> >>>> 102 263 420 381
> >>>> 154 547 672 239 2215
> >>>> 117 347 389 383
> >>>> 165 688 683 397 2380
> >>>> 205 364 400 393
> >>>> 561 1340 1024 995 2537
> >>>> 777 637 899 775
> >>>> 762 1766 1336 1903 2890
> >>>> 1458 1248 1218 1051
> >>>> 1162 3464 1970 4859 3602
> >>>> 3405 1702 1718 1712
> >>>> 1761 4666 3274 10928 8028
> >>>> 7927 2780 3249 2691
> >>>> 2081 6777 2993 15066 11008
> >>>> 10864 3595 3281 2859
> >>>> 4241 23038 Could not transfer (Java
> client
> >>>> Broken) 52853 33351 32957 16234
> >>>> 115777 7991
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 5/10/06, *Samisa Abeysinghe* <
> samisa.abeysinghe@gmail.com <ma...@gmail.com>
> >>>> <mailto: samisa.abeysinghe@gmail.com
> <ma...@gmail.com>>> wrote:
> >>>>
> >>>> Well, from this I am certain that we should not
> change the
> >>>> buffer size
> >>>> to 1024*1024, rather keep it at 1024.
> >>>> Please try with stat, using a dynamic buffer size
> based on the
> >>>> file
> >>>> size, my gut feel is that it is going to be the best
> solution.
> >>>>
> >>>> Thanks,
> >>>> Samisa...
> >>>>
> >>>> Kapila Dissanayake wrote:
> >>>>
> >>>> >
> >>>> > Hi,
> >>>> >
> >>>> > I did some testing agian with to measure the binary
> file
> >>>> transfer
> >>>> > times. In axis2 c data_handler.c, different buffer
> sizes
> >>>> (1024 and
> >>>> > 1024*1024) were used to read files.
> >>>> >
> >>>> > I think using fixed size buffers to read binary
> files in axis2-C
> >>>> has
> >>>> > been affected in some cases to slow down the transfer.
> >>>> >
> >>>> > Thanks
> >>>> >
> >>>> > Kapila
> >>>> >
> >>>> > Please see the following test results
> >>>> >
> >>>> > Environment
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > ----------
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > Processor - Intel Celeron M 1.40GHz
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > RAM - 256 MB
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > OS - Fedora Core 4.
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > Java Server - axis2 Java
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > C Server - axis2
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > mtom C and java test clients were used
>
> ...
>
> [Message clipped]
>
>
>
>
> --
> "May the SourcE be with u"
> http://webservices.apache.org/~thilina/
> <http://webservices.apache.org/%7Ethilina/>
> http://thilinag.blogspot.com/
> http://www.bloglines.com/blog/Thilina
Re: [AXIS2] MTOM performance Measuring - http_transport_utils.c modified
Posted by Samisa Abeysinghe <sa...@gmail.com>.
Thilina Gunarathne wrote:
> Wow Wow... pretty impressive.. Way to go guys... Keep up the good work..
>
> It'll be great if we can give it a try to benchmark our stuff against
> WCF ,Xfire , etc too...
>
> Have you already implemented the Base64 conversion to happen
> automatically when the binary is not optimized.. It's critical to have
> it before we claim about the XOP/MTOM feature.
I implemented the base64 support for non optimized case today.
Multiple attachments too work.
Thanks,
Samisa...
>
> ~Thilina
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Re: [AXIS2] MTOM performance Measuring - http_transport_utils.c modified
Posted by Thilina Gunarathne <cs...@gmail.com>.
Wow Wow... pretty impressive.. Way to go guys... Keep up the good work..
It'll be great if we can give it a try to benchmark our stuff against WCF
,Xfire , etc too...
Have you already implemented the Base64 conversion to happen
automatically when the binary is not optimized.. It's critical to have it
before we claim about the XOP/MTOM feature.
~Thilina
On 5/13/06, Kapila Dissanayake <ka...@gmail.com> wrote:
>
> Hi ,
>
> I did another modification to improve the messege transfer speed.
>
> http_transport_utils.c was modified. (Buffer size was changed from 1024 to
> 1024 * 1024)
>
> Now the transfer speed has been changed significantly. Please see the
> table below.
>
>
> The attached file includes the svn diff of the change.
>
> Thank You
>
> Kapila
>
>
> (File Size) vs (Time Taken in milli seconds) to send binary
> attachments Environment ---------- Processor -
> Intel Celeron M 1.40GHz RAM - 256 MB OS - Fedora Core 4. Java
> Server - axis2 Java C Server - axis2 mtom C and java test
> clients were used http_transport_utils.c was modified. (Buffer size
> was changed from 1024 to 1024 * 1024) Time Taken in milli seconds File
> Size (kb) Java Server Java Server C Server C Server C Server C
> Server C Server C Server C Server C Server Java Client (ms) C
> Client (ms) C Client (ms) Java Client (ms) *data_handler.c
> Buffer Size 1024 * *data_handler.c 'Buffer Size 1024 X 1024 * *Using stat*
> *http_transport_utils.c 'Buffer Size 1024 X 1024 * * * *Buffer Size '1024
> * *Buffer Size '1024 X 1024 * *Using stat* *http_transport_utils.c 'Buffer
> Size 1024 X 1024 * ----------------- ----------------- -----------------
> ----------------- ----------------- ----------------- -----------------
> ----------------- ----------------- ---------------------------------
> ----------------- ----------------- ----------------- ----------------- ---------------------------------
> 13.8 147 158 73 1884 78 76 188 143 136 136 28.2 515 488 78
> 1912 78 78 222 413 364 387 37.8 538 694 131 1917 81 81 232 474
> 377 396 132 571 807 212 2177 102 83 263 420 381 432 154 547
> 672 239 2215 117 84 347 389 383 517 165 688 683 397 2380 205 85
> 364 400 393 602 561 1340 1024 995 2537 777 100 637 899 775 766
> 762 1766 1336 1903 2890 1458 111 1248 1218 1051 1055 1162 3464
> 1970 4859 3602 3405 135 1702 1718 1712 1092 1761 4666 3274
> 10928 8028 7927 188 2780 3249 2691 2596 2081 6777 2993 15066
> 11008 10864 219 3595 3281 2859 3185 4241 23038 Could not transfer
> (Java client Broken) 52853 33351 32957 519
> 16234 115777 7991 6499
>
>
> On 5/12/06, Samisa Abeysinghe <sa...@gmail.com> wrote:
> >
> > BTW,
> > If we are going to use chunking at some point in time, then the data
> > handler logic for reading file too has to change to keep track of the
> > amount of chunks already read, and current position of read pointer.
> > There is no provision for such reading in the code as of now, neither in
> > the data handler nor in the mime_output that uses the data handler API.
> > So there is considerable amount of work on the part of the data hander
> > and the places that uses the API of data handler to read to turn on
> > chunking as far as I can see.
> > Thanks,
> > Samisa...
> >
> > Samisa Abeysinghe wrote:
> >
> > > Nabeel wrote:
> > >
> > >> Kapila,
> > >> With your latest changes you always read the file in one go
> > >> (i.e. not in chunks) and you don't need a loop. The attached compact
> > >> code does the same thing.
> > >>
> > >> Suggestion:
> > >> When we need to attach *very* large file, we may not be keep the
> > >> whole file in memory. IMO, this where chunk reading is really useful.
> > >> If you are to implement this, you'll have to change the code from
> > >> data_handler -> mime_body_part -> mime_output to om_output so that
> > >> you'll keep only chunk_size amount of bytes in memory at any given
> > time.
> > >
> > >
> > > Chunking is a very good idea, in fact that is the perfect solution for
> >
> > > large attachments.
> > > However, AFAIK, we have chunking capability only on the client side
> > > transport, not on the server side transport.
> > > Hence, we got to implement chunking capability into simple axis server
> >
> > > transport first. (May be we can raise a Jira on this). Or we have to
> > > see if we can piggyback on Apache for chunking.
> > > Secondly, we got to research and find out the ideal chunk size. As we
> > > have seen through the results sent by Kapila, too small a chunk size
> > > makes too large file transmission very slow and too large chunk size
> > > makes small file transmission very slow. Best is to not use chunking
> > > for small files and use chunking, with a reasonable chunk size, for
> > > large files. So we have several parameters to figure out.
> > > i. Max size we can read a file in one go without chunking.
> > > ii. When chunking is turned on, ideal chunk size.
> > > As I mentioned earlier, we need some research around this to decide on
> >
> > > the figures.
> > >
> > > Thanks,
> > > Samisa...
> > >
> > >>
> > >> -Nabeel
> > >>
> > >> Samisa Abeysinghe wrote:
> > >>
> > >>> Cool, so there sure is an improvement when using stat on read side
> > >>> of data handler. Hence lets use this solution.
> > >>> Still we need to figure out what goes wrong with large files beyond
> > 1M
> > >>>
> > >>> Samisa...
> > >>>
> > >>> Kapila Dissanayake wrote:
> > >>>
> > >>>> Hi
> > >>>>
> > >>>> I did some modification to the data handler. Insted of using fixed
> > >>>> buffer size to read files
> > >>>> 'stat' is used.
> > >>>>
> > >>>> Please see the perfomance details.
> > >>>>
> > >>>> Thank You
> > >>>>
> > >>>> Kapila
> > >>>>
> > >>>>
> > >>>> Environment ---------- Processor - Intel
> > >>>> Celeron M 1.40GHz
> > >>>> RAM - 256 MB OS - Fedora Core 4. Java Server -
> > >>>> axis2 Java C Server - axis2 mtom C and java test
> > >>>> clients were used (in axis2 c data_handler.c, different buffer
> > >>>> sizes were used to read files)
> > >>>> Time Taken in milli seconds
> > >>>> File Size (kb) Java Server (ms) Java Server
> > >>>> (ms) C Server (ms) C Server (ms)
> > >>>> Java Client (ms) C Client (ms) C
> > >>>> Client (ms) Java Client (ms)
> > >>>> *Buffer Size 1024 * *Buffer
> >
> > >>>> Size 1024 X 1024 * *Using stat* * * *Buffer Size '1024
> > >>>> * *Buffer Size '1024 X 1024 * *Using stat *
> > >>>> ----------------- ----------------- -----------------
> > >>>> ----------------- ----------------- -----------------
> > >>>> ----------------- ----------------- -----------------
> > >>>> ----------------- ----------------- -----------------
> > >>>> -----------------
> > >>>> 13.8 147 158 73 1884
> > >>>> 78 188 143 136
> > >>>> 28.2 515 488 78 1912
> > >>>> 78 222 413 364
> > >>>> 37.8 538 694 131 1917
> > >>>> 81 232 474 377
> > >>>> 132 571 807 212 2177
> > >>>> 102 263 420 381
> > >>>> 154 547 672 239 2215
> > >>>> 117 347 389 383
> > >>>> 165 688 683 397 2380
> > >>>> 205 364 400 393
> > >>>> 561 1340 1024 995 2537
> > >>>> 777 637 899 775
> > >>>> 762 1766 1336 1903 2890
> > >>>> 1458 1248 1218 1051
> > >>>> 1162 3464 1970 4859 3602
> > >>>> 3405 1702 1718 1712
> > >>>> 1761 4666 3274 10928 8028
> > >>>> 7927 2780 3249 2691
> > >>>> 2081 6777 2993 15066 11008
> > >>>> 10864 3595 3281 2859
> > >>>> 4241 23038 Could not transfer (Java client
> > >>>> Broken) 52853 33351 32957 16234
> > >>>> 115777 7991
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On 5/10/06, *Samisa Abeysinghe* < samisa.abeysinghe@gmail.com
> > >>>> <ma...@gmail.com>> wrote:
> > >>>>
> > >>>> Well, from this I am certain that we should not change the
> > >>>> buffer size
> > >>>> to 1024*1024, rather keep it at 1024.
> > >>>> Please try with stat, using a dynamic buffer size based on the
> > >>>> file
> > >>>> size, my gut feel is that it is going to be the best solution.
> > >>>>
> > >>>> Thanks,
> > >>>> Samisa...
> > >>>>
> > >>>> Kapila Dissanayake wrote:
> > >>>>
> > >>>> >
> > >>>> > Hi,
> > >>>> >
> > >>>> > I did some testing agian with to measure the binary file
> > >>>> transfer
> > >>>> > times. In axis2 c data_handler.c, different buffer sizes
> > >>>> (1024 and
> > >>>> > 1024*1024) were used to read files.
> > >>>> >
> > >>>> > I think using fixed size buffers to read binary files in
> > axis2-C
> > >>>> has
> > >>>> > been affected in some cases to slow down the transfer.
> > >>>> >
> > >>>> > Thanks
> > >>>> >
> > >>>> > Kapila
> > >>>> >
> > >>>> > Please see the following test results
> > >>>> >
> > >>>> > Environment
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > ----------
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > Processor - Intel Celeron M 1.40GHz
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > RAM - 256 MB
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > OS - Fedora Core 4.
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > Java Server - axis2 Java
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > C Server - axis2
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > mtom C and java test clients were used
>
> ...
>
> [Message clipped]
>
--
"May the SourcE be with u"
http://webservices.apache.org/~thilina/
http://thilinag.blogspot.com/
http://www.bloglines.com/blog/Thilina