You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-dev@axis.apache.org by "Rich Scheuerle (JIRA)" <ji...@apache.org> on 2007/07/11 20:43:04 UTC
[jira] Created: (AXIS2-2946) Attachment Perf Improvement
Attachment Perf Improvement
---------------------------
Key: AXIS2-2946
URL: https://issues.apache.org/jira/browse/AXIS2-2946
Project: Axis 2.0 (Axis2)
Issue Type: Improvement
Components: kernel
Reporter: Rich Scheuerle
Assignee: Rich Scheuerle
I am working on a performance improvement improvement reading attachments.
Currently the Axiom Attachments code reads bytes until it surpasses the file threshold. If the file threshold is high (i.e. 100M) this causes a very large buffer to be created....even if the attachment is stored on file.
The improvement is to use the content-length of the message as a heuristic to determine if the attachments should be flushed to file.
This improves performance and memory footprint.
The first part of the change is to add an Axiom Attachment constructor to axiom that accepts a content-length parameter. If the content-length is set, then the the heuristic is used.
The second part of the change is to add code to the Axis2 kernel builder to pass in the content-length.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
Re: [jira] Created: (AXIS2-2946) Attachment Perf Improvement
Posted by Sanjiva Weerawarana <sa...@opensource.lk>.
Rich Scheuerle (JIRA) wrote:
> Attachment Perf Improvement ---------------------------
>
> Key: AXIS2-2946 URL: https://issues.apache.org/jira/browse/AXIS2-2946
> Project: Axis 2.0 (Axis2) Issue Type: Improvement Components: kernel
> Reporter: Rich Scheuerle Assignee: Rich Scheuerle
>
>
> I am working on a performance improvement improvement reading
> attachments.
>
> Currently the Axiom Attachments code reads bytes until it surpasses the
> file threshold. If the file threshold is high (i.e. 100M) this causes
> a very large buffer to be created....even if the attachment is stored
> on file.
Isn't this user config error?
> The improvement is to use the content-length of the message as a
> heuristic to determine if the attachments should be flushed to file.
> This improves performance and memory footprint.
In most cases with attachments the messages will be chunked. There's no
content length available in such cases to make the test up front on
whether to write to a file or not. Sending a 100M or even 10M attachment
without HTTP chunking is simply terrible form IMO.
Anyway if that's the edge case that you want to handle I'm fine with it. I
assume this'll go in post 1.3 due to changes to kernel APIs.
Sanjiva.
--
Sanjiva Weerawarana, Ph.D.
Founder & Director; Lanka Software Foundation; http://www.opensource.lk/
Founder, Chairman & CEO; WSO2, Inc.; http://www.wso2.com/
Director; Open Source Initiative; http://www.opensource.org/
Member; Apache Software Foundation; http://www.apache.org/
Visiting Lecturer; University of Moratuwa; http://www.cse.mrt.ac.lk/
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Resolved: (AXIS2-2946) Attachment Perf Improvement
Posted by "Rich Scheuerle (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rich Scheuerle resolved AXIS2-2946.
-----------------------------------
Resolution: Fixed
Committed the Axis2 BuilderUtils change. The revision is 555611.
The result will be improved performance and reduced memory footprint when there are large attachments and a large fileThreshold.
Thanks!
Rich
> Attachment Perf Improvement
> ---------------------------
>
> Key: AXIS2-2946
> URL: https://issues.apache.org/jira/browse/AXIS2-2946
> Project: Axis 2.0 (Axis2)
> Issue Type: Improvement
> Components: kernel
> Reporter: Rich Scheuerle
> Assignee: Rich Scheuerle
>
> I am working on a performance improvement improvement reading attachments.
> Currently the Axiom Attachments code reads bytes until it surpasses the file threshold. If the file threshold is high (i.e. 100M) this causes a very large buffer to be created....even if the attachment is stored on file.
> The improvement is to use the content-length of the message as a heuristic to determine if the attachments should be flushed to file.
> This improves performance and memory footprint.
> The first part of the change is to add an Axiom Attachment constructor to axiom that accepts a content-length parameter. If the content-length is set, then the the heuristic is used.
> The second part of the change is to add code to the Axis2 kernel builder to pass in the content-length.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Resolved: (AXIS2-2946) Attachment Perf Improvement
Posted by "Rich Scheuerle (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rich Scheuerle resolved AXIS2-2946.
-----------------------------------
Resolution: Fixed
REV=555743
> Attachment Perf Improvement
> ---------------------------
>
> Key: AXIS2-2946
> URL: https://issues.apache.org/jira/browse/AXIS2-2946
> Project: Axis 2.0 (Axis2)
> Issue Type: Improvement
> Components: kernel
> Reporter: Rich Scheuerle
> Assignee: Rich Scheuerle
>
> I am working on a performance improvement improvement reading attachments.
> Currently the Axiom Attachments code reads bytes until it surpasses the file threshold. If the file threshold is high (i.e. 100M) this causes a very large buffer to be created....even if the attachment is stored on file.
> The improvement is to use the content-length of the message as a heuristic to determine if the attachments should be flushed to file.
> This improves performance and memory footprint.
> The first part of the change is to add an Axiom Attachment constructor to axiom that accepts a content-length parameter. If the content-length is set, then the the heuristic is used.
> The second part of the change is to add code to the Axis2 kernel builder to pass in the content-length.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Reopened: (AXIS2-2946) Attachment Perf Improvement
Posted by "Rich Scheuerle (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rich Scheuerle reopened AXIS2-2946:
-----------------------------------
As part of this fix, I replaced the following code:
do {
int len;
int off = 0;
int rem = fileStorageThreshold;
while ((len = partStream.read(buffer, off, rem)) > 0) {
off = off + len;
rem = rem - len;
}
count += off;
} while (partStream.available() > 0);
with
count = partStream.read(buffer).
The "old code" is definitely wrong. Notice that the buffer is over-written if the while loop is repeated.
However the "new code" could possibly fail if the underlying stream delivers the bytes in chunks (not sure if that could actually happen).
I am testing a fix right now that uses the following logic:
/**
* Read bytes into the buffer until full or until the EOS
* @param is
* @param buffer
* @return number of bytes read
* @throws IOException
*/
private static int readToBuffer(InputStream is, byte[] buffer) throws IOException {
int index = 0;
int remainder = buffer.length;
do {
int bytesRead;
while ((bytesRead = is.read(buffer, index, remainder)) > 0) {
index += bytesRead;
remainder -= bytesRead;
}
} while (remainder > 0 && is.available() > 0); // repeat if more bytes are now available
return index;
}
Kudos to Thilina for reviewing this patch.
> Attachment Perf Improvement
> ---------------------------
>
> Key: AXIS2-2946
> URL: https://issues.apache.org/jira/browse/AXIS2-2946
> Project: Axis 2.0 (Axis2)
> Issue Type: Improvement
> Components: kernel
> Reporter: Rich Scheuerle
> Assignee: Rich Scheuerle
>
> I am working on a performance improvement improvement reading attachments.
> Currently the Axiom Attachments code reads bytes until it surpasses the file threshold. If the file threshold is high (i.e. 100M) this causes a very large buffer to be created....even if the attachment is stored on file.
> The improvement is to use the content-length of the message as a heuristic to determine if the attachments should be flushed to file.
> This improves performance and memory footprint.
> The first part of the change is to add an Axiom Attachment constructor to axiom that accepts a content-length parameter. If the content-length is set, then the the heuristic is used.
> The second part of the change is to add code to the Axis2 kernel builder to pass in the content-length.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org
[jira] Commented: (AXIS2-2946) Attachment Perf Improvement
Posted by "Rich Scheuerle (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12511842 ]
Rich Scheuerle commented on AXIS2-2946:
---------------------------------------
I added the Axiom changes. The revision is 555379.
I will add the Axis2 changes on Thursday.
Thanks,
Rich
> Attachment Perf Improvement
> ---------------------------
>
> Key: AXIS2-2946
> URL: https://issues.apache.org/jira/browse/AXIS2-2946
> Project: Axis 2.0 (Axis2)
> Issue Type: Improvement
> Components: kernel
> Reporter: Rich Scheuerle
> Assignee: Rich Scheuerle
>
> I am working on a performance improvement improvement reading attachments.
> Currently the Axiom Attachments code reads bytes until it surpasses the file threshold. If the file threshold is high (i.e. 100M) this causes a very large buffer to be created....even if the attachment is stored on file.
> The improvement is to use the content-length of the message as a heuristic to determine if the attachments should be flushed to file.
> This improves performance and memory footprint.
> The first part of the change is to add an Axiom Attachment constructor to axiom that accepts a content-length parameter. If the content-length is set, then the the heuristic is used.
> The second part of the change is to add code to the Axis2 kernel builder to pass in the content-length.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-dev-help@ws.apache.org