You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commons-dev@ws.apache.org by "Michal Stochmialek (JIRA)" <ji...@apache.org> on 2007/09/13 11:59:32 UTC
[jira] Created: (WSCOMMONS-247) choosing attachment file caching
strategy is broken
choosing attachment file caching strategy is broken
---------------------------------------------------
Key: WSCOMMONS-247
URL: https://issues.apache.org/jira/browse/WSCOMMONS-247
Project: WS-Commons
Issue Type: Bug
Components: AXIOM
Reporter: Michal Stochmialek
The bug is a gap in conditions while choosing file caching strategy:
if (contentLength != 0 &&
contentLength <= fileStorageThreshold) {
// Since the content-length is less than the threshold, we can safely
// store it in memory.
// [... creating a Part in memory ...]
} else if (contentLength != 0 &&
contentLength > fileStorageThreshold * 2) {
// The content-length is much bigger than the threshold, then always
// store the attachments in a file. This prevents unnecessary buffering.
// [... creating a port in file ...]
} else {
// Read chunks of data to determine the size
// of the attachment. This can wasteful because we need to gc the buffers.
// [... determining the size by reading the beginning of stream
}
Those conditions works fine for:
- contentLength != 0 and contentLenght <= threshold
- contentLength != 0 and contentLenght > threshold * 2
- contentLength == 0
but not for:
- contentLength != 0 and threshold < contentLenght > threshold * 2
In this case, the flow should go to the first condition, not the third one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: commons-dev-help@ws.apache.org
[jira] Commented: (WSCOMMONS-247) choosing attachment file caching
strategy is broken
Posted by "Rich Scheuerle (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WSCOMMONS-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12527089 ]
Rich Scheuerle commented on WSCOMMONS-247:
------------------------------------------
Michel,
A couple key points.
1) The contentLength in this part of the code is the contentLength of the entire message.
2) We are using the contentLength of the entire message as a heuristic to avoid unnecessary buffering.
So the idea is, if the contentLength of the whole message is much greater than the fileStorageThreshold (for an attachment)
then just assume that the attachment should be put in a file.
For the case that you question, the contentLength of the message > fileStorageThreshold but < fileStorageThreshold*2.
This might be the case where we have a large soap part and several medium sized attachments.
Under these conditions we correctly fall into category (3) and determine whether to create a PartOnMemory or PartOnFile.
-----------------
We do need changes to the heuristic and or loading code. More work is needed.
Here are is an alternative suggestion:
Right now the code makes a decision to make a PartOnFile or PartOnMemory. It would be nice if we had
a DynamicPart. The implementation of DynamicPart could keep the bytes in memory until the fileStorageThreshold was hit and then
silently convert to putting the bytes on file. This kind of mechanism would allow us to avoid heuristics and buffering.
It would also work better in conditions where the incoming message is not chunked (i.e. there is no contentLength).
I suggest that you close this issue, or help provide a more longterm solution.
Thanks,
Scheu
> choosing attachment file caching strategy is broken
> ---------------------------------------------------
>
> Key: WSCOMMONS-247
> URL: https://issues.apache.org/jira/browse/WSCOMMONS-247
> Project: WS-Commons
> Issue Type: Bug
> Components: AXIOM
> Reporter: Michal Stochmialek
>
> The bug is a gap in conditions while choosing file caching strategy:
> if (contentLength != 0 &&
> contentLength <= fileStorageThreshold) {
> // Since the content-length is less than the threshold, we can safely
> // store it in memory.
> // [... creating a Part in memory ...]
> } else if (contentLength != 0 &&
> contentLength > fileStorageThreshold * 2) {
> // The content-length is much bigger than the threshold, then always
> // store the attachments in a file. This prevents unnecessary buffering.
> // [... creating a port in file ...]
>
> } else {
> // Read chunks of data to determine the size
> // of the attachment. This can wasteful because we need to gc the buffers.
> // [... determining the size by reading the beginning of stream
> }
> Those conditions works fine for:
> - contentLength != 0 and contentLenght <= threshold
> - contentLength != 0 and contentLenght > threshold * 2
> - contentLength == 0
> but not for:
> - contentLength != 0 and threshold < contentLenght > threshold * 2
> In this case, the flow should go to the first condition, not the third one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: commons-dev-help@ws.apache.org
[jira] Assigned: (WSCOMMONS-247) choosing attachment file caching
strategy is broken
Posted by "Rich Scheuerle (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WSCOMMONS-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rich Scheuerle reassigned WSCOMMONS-247:
----------------------------------------
Assignee: Rich Scheuerle
> choosing attachment file caching strategy is broken
> ---------------------------------------------------
>
> Key: WSCOMMONS-247
> URL: https://issues.apache.org/jira/browse/WSCOMMONS-247
> Project: WS-Commons
> Issue Type: Bug
> Components: AXIOM
> Reporter: Michal Stochmialek
> Assignee: Rich Scheuerle
>
> The bug is a gap in conditions while choosing file caching strategy:
> if (contentLength != 0 &&
> contentLength <= fileStorageThreshold) {
> // Since the content-length is less than the threshold, we can safely
> // store it in memory.
> // [... creating a Part in memory ...]
> } else if (contentLength != 0 &&
> contentLength > fileStorageThreshold * 2) {
> // The content-length is much bigger than the threshold, then always
> // store the attachments in a file. This prevents unnecessary buffering.
> // [... creating a port in file ...]
>
> } else {
> // Read chunks of data to determine the size
> // of the attachment. This can wasteful because we need to gc the buffers.
> // [... determining the size by reading the beginning of stream
> }
> Those conditions works fine for:
> - contentLength != 0 and contentLenght <= threshold
> - contentLength != 0 and contentLenght > threshold * 2
> - contentLength == 0
> but not for:
> - contentLength != 0 and threshold < contentLenght > threshold * 2
> In this case, the flow should go to the first condition, not the third one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: commons-dev-help@ws.apache.org
[jira] Issue Comment Edited: (WSCOMMONS-247) choosing attachment
file caching strategy is broken
Posted by "Michal Stochmialek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WSCOMMONS-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12527050 ]
michal.stochmialek edited comment on WSCOMMONS-247 at 9/13/07 3:00 AM:
-----------------------------------------------------------------------
Of course the patch should look something like this:
if (contentLength != 0 &&
- contentLength <= fileStorageThreshold) {
+ contentLength <= fileStorageThreshold * 2) {
// Since the content-length is less than the threshold, we can safely
was (Author: michal.stochmialek):
Of course that patch should look something like this:
if (contentLength != 0 &&
- contentLength <= fileStorageThreshold) {
+ contentLength <= fileStorageThreshold * 2) {
// Since the content-length is less than the threshold, we can safely
> choosing attachment file caching strategy is broken
> ---------------------------------------------------
>
> Key: WSCOMMONS-247
> URL: https://issues.apache.org/jira/browse/WSCOMMONS-247
> Project: WS-Commons
> Issue Type: Bug
> Components: AXIOM
> Reporter: Michal Stochmialek
>
> The bug is a gap in conditions while choosing file caching strategy:
> if (contentLength != 0 &&
> contentLength <= fileStorageThreshold) {
> // Since the content-length is less than the threshold, we can safely
> // store it in memory.
> // [... creating a Part in memory ...]
> } else if (contentLength != 0 &&
> contentLength > fileStorageThreshold * 2) {
> // The content-length is much bigger than the threshold, then always
> // store the attachments in a file. This prevents unnecessary buffering.
> // [... creating a port in file ...]
>
> } else {
> // Read chunks of data to determine the size
> // of the attachment. This can wasteful because we need to gc the buffers.
> // [... determining the size by reading the beginning of stream
> }
> Those conditions works fine for:
> - contentLength != 0 and contentLenght <= threshold
> - contentLength != 0 and contentLenght > threshold * 2
> - contentLength == 0
> but not for:
> - contentLength != 0 and threshold < contentLenght > threshold * 2
> In this case, the flow should go to the first condition, not the third one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: commons-dev-help@ws.apache.org
[jira] Commented: (WSCOMMONS-247) choosing attachment file caching
strategy is broken
Posted by "Thilina Gunarathne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WSCOMMONS-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12527102 ]
Thilina Gunarathne commented on WSCOMMONS-247:
----------------------------------------------
Hi Scheu,
>Right now the code makes a decision to make a PartOnFile or PartOnMemory. It would be nice if we had
>a DynamicPart. The implementation of DynamicPart could keep the bytes in memory until the fileStorageThreshold was hit and then
>silently convert to putting the bytes on file.
Though the decision happens outside of the parts, effectively the behaviour was supposed to be what you are talking about. We buffer the bytes in memory until some threshold occurs and then changes to putting the contents in to a PartOnFile if it exceeds the limit. If not the buffer will be parsed to the PartOnMemory.
I see the dynamic part you are proposing only as moving the above code bit in to that from the current place..
Anyway not sure whether I got you correct. Correct me if I'm misunderstanding the proposal.
> choosing attachment file caching strategy is broken
> ---------------------------------------------------
>
> Key: WSCOMMONS-247
> URL: https://issues.apache.org/jira/browse/WSCOMMONS-247
> Project: WS-Commons
> Issue Type: Bug
> Components: AXIOM
> Reporter: Michal Stochmialek
> Assignee: Rich Scheuerle
>
> The bug is a gap in conditions while choosing file caching strategy:
> if (contentLength != 0 &&
> contentLength <= fileStorageThreshold) {
> // Since the content-length is less than the threshold, we can safely
> // store it in memory.
> // [... creating a Part in memory ...]
> } else if (contentLength != 0 &&
> contentLength > fileStorageThreshold * 2) {
> // The content-length is much bigger than the threshold, then always
> // store the attachments in a file. This prevents unnecessary buffering.
> // [... creating a port in file ...]
>
> } else {
> // Read chunks of data to determine the size
> // of the attachment. This can wasteful because we need to gc the buffers.
> // [... determining the size by reading the beginning of stream
> }
> Those conditions works fine for:
> - contentLength != 0 and contentLenght <= threshold
> - contentLength != 0 and contentLenght > threshold * 2
> - contentLength == 0
> but not for:
> - contentLength != 0 and threshold < contentLenght > threshold * 2
> In this case, the flow should go to the first condition, not the third one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: commons-dev-help@ws.apache.org
[jira] Closed: (WSCOMMONS-247) choosing attachment file caching
strategy is broken
Posted by "Michal Stochmialek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WSCOMMONS-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michal Stochmialek closed WSCOMMONS-247.
----------------------------------------
Resolution: Invalid
I'll close the issue for now, since the "condition gap" was just misunderstanding
of current implementation. Although I like your suggestion, Scheu. And in my spare time
I'll try to implement it (unless Thilina was right).
> choosing attachment file caching strategy is broken
> ---------------------------------------------------
>
> Key: WSCOMMONS-247
> URL: https://issues.apache.org/jira/browse/WSCOMMONS-247
> Project: WS-Commons
> Issue Type: Bug
> Components: AXIOM
> Reporter: Michal Stochmialek
> Assignee: Rich Scheuerle
>
> The bug is a gap in conditions while choosing file caching strategy:
> if (contentLength != 0 &&
> contentLength <= fileStorageThreshold) {
> // Since the content-length is less than the threshold, we can safely
> // store it in memory.
> // [... creating a Part in memory ...]
> } else if (contentLength != 0 &&
> contentLength > fileStorageThreshold * 2) {
> // The content-length is much bigger than the threshold, then always
> // store the attachments in a file. This prevents unnecessary buffering.
> // [... creating a port in file ...]
>
> } else {
> // Read chunks of data to determine the size
> // of the attachment. This can wasteful because we need to gc the buffers.
> // [... determining the size by reading the beginning of stream
> }
> Those conditions works fine for:
> - contentLength != 0 and contentLenght <= threshold
> - contentLength != 0 and contentLenght > threshold * 2
> - contentLength == 0
> but not for:
> - contentLength != 0 and threshold < contentLenght > threshold * 2
> In this case, the flow should go to the first condition, not the third one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: commons-dev-help@ws.apache.org
[jira] Commented: (WSCOMMONS-247) choosing attachment file caching
strategy is broken
Posted by "Michal Stochmialek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WSCOMMONS-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12527050 ]
Michal Stochmialek commented on WSCOMMONS-247:
----------------------------------------------
Of course that patch should look something like this:
if (contentLength != 0 &&
- contentLength <= fileStorageThreshold) {
+ contentLength <= fileStorageThreshold * 2) {
// Since the content-length is less than the threshold, we can safely
> choosing attachment file caching strategy is broken
> ---------------------------------------------------
>
> Key: WSCOMMONS-247
> URL: https://issues.apache.org/jira/browse/WSCOMMONS-247
> Project: WS-Commons
> Issue Type: Bug
> Components: AXIOM
> Reporter: Michal Stochmialek
>
> The bug is a gap in conditions while choosing file caching strategy:
> if (contentLength != 0 &&
> contentLength <= fileStorageThreshold) {
> // Since the content-length is less than the threshold, we can safely
> // store it in memory.
> // [... creating a Part in memory ...]
> } else if (contentLength != 0 &&
> contentLength > fileStorageThreshold * 2) {
> // The content-length is much bigger than the threshold, then always
> // store the attachments in a file. This prevents unnecessary buffering.
> // [... creating a port in file ...]
>
> } else {
> // Read chunks of data to determine the size
> // of the attachment. This can wasteful because we need to gc the buffers.
> // [... determining the size by reading the beginning of stream
> }
> Those conditions works fine for:
> - contentLength != 0 and contentLenght <= threshold
> - contentLength != 0 and contentLenght > threshold * 2
> - contentLength == 0
> but not for:
> - contentLength != 0 and threshold < contentLenght > threshold * 2
> In this case, the flow should go to the first condition, not the third one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: commons-dev-help@ws.apache.org