You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by "Emmanuel Lecharny (JIRA)" <ji...@apache.org> on 2006/09/07 08:04:22 UTC
[jira] Created: (DIRSERVER-731) Streming big objects
Streming big objects
--------------------
Key: DIRSERVER-731
URL: http://issues.apache.org/jira/browse/DIRSERVER-731
Project: Directory ApacheDS
Issue Type: Improvement
Reporter: Emmanuel Lecharny
Priority: Critical
We really need to implement a streming of big objects, like JpegPhoto, otherwise the server can be killed trying to manage them.
The problem is that they are stored entirely in memory, and this is not an option for megabytes of data in a JVM. We should find a way to work with MINA to send bytes on the fly (for instance by packets of 8kb). The very same when we receive huge data : we should store them somwhere on the disk instead of creating a byte[] large enough to store the data.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DIRSERVER-731) Streaming big objects
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Lecharny updated DIRSERVER-731:
----------------------------------------
Fix Version/s: (was: 1.5.1)
1.5.2
Far too complex to be implemented in 1.5.1 release. Postponed ...
> Streaming big objects
> ---------------------
>
> Key: DIRSERVER-731
> URL: https://issues.apache.org/jira/browse/DIRSERVER-731
> Project: Directory ApacheDS
> Issue Type: Improvement
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 1.5.2
>
>
> We really need to implement a streming of big objects, like JpegPhoto, otherwise the server can be killed trying to manage them.
> The problem is that they are stored entirely in memory, and this is not an option for megabytes of data in a JVM. We should find a way to work with MINA to send bytes on the fly (for instance by packets of 8kb). The very same when we receive huge data : we should store them somwhere on the disk instead of creating a byte[] large enough to store the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DIRSERVER-731) Streaming big objects
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Lecharny updated DIRSERVER-731:
----------------------------------------
Fix Version/s: (was: 1.5.3)
2.0.0
Postponed
> Streaming big objects
> ---------------------
>
> Key: DIRSERVER-731
> URL: https://issues.apache.org/jira/browse/DIRSERVER-731
> Project: Directory ApacheDS
> Issue Type: Improvement
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0
>
>
> We really need to implement a streming of big objects, like JpegPhoto, otherwise the server can be killed trying to manage them.
> The problem is that they are stored entirely in memory, and this is not an option for megabytes of data in a JVM. We should find a way to work with MINA to send bytes on the fly (for instance by packets of 8kb). The very same when we receive huge data : we should store them somwhere on the disk instead of creating a byte[] large enough to store the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DIRSERVER-731) Streaming big objects
Posted by "Alex Karasulu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Karasulu updated DIRSERVER-731:
------------------------------------
Fix Version/s: (was: 1.5.2)
1.5.3
Postpone again. Until this is applied the server will be prone to DoS attacks.
> Streaming big objects
> ---------------------
>
> Key: DIRSERVER-731
> URL: https://issues.apache.org/jira/browse/DIRSERVER-731
> Project: Directory ApacheDS
> Issue Type: Improvement
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 1.5.3
>
>
> We really need to implement a streming of big objects, like JpegPhoto, otherwise the server can be killed trying to manage them.
> The problem is that they are stored entirely in memory, and this is not an option for megabytes of data in a JVM. We should find a way to work with MINA to send bytes on the fly (for instance by packets of 8kb). The very same when we receive huge data : we should store them somwhere on the disk instead of creating a byte[] large enough to store the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-731) Streaming big objects
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12520813 ]
Emmanuel Lecharny commented on DIRSERVER-731:
---------------------------------------------
Well, I have thought about implementing such a workaround, and it's not that big (there is one place where we can do that in the code), but this won't solve any DoS attack, as we must accept big objects like JpegPhoto. Those kind of object, even if limited to a few hundred of Kb (!), will still be stored in memory, and can kill the server fast.
However, we can implement this workaround first (let's say it can be done, tests included, in one day), and start to think about streaming data for 2.0 (I have started a branch for that, but it's a little bit too old now ...)
> Streaming big objects
> ---------------------
>
> Key: DIRSERVER-731
> URL: https://issues.apache.org/jira/browse/DIRSERVER-731
> Project: Directory ApacheDS
> Issue Type: Improvement
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 1.5.1
>
>
> We really need to implement a streming of big objects, like JpegPhoto, otherwise the server can be killed trying to manage them.
> The problem is that they are stored entirely in memory, and this is not an option for megabytes of data in a JVM. We should find a way to work with MINA to send bytes on the fly (for instance by packets of 8kb). The very same when we receive huge data : we should store them somwhere on the disk instead of creating a byte[] large enough to store the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DIRSERVER-731) Streaming big objects
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Lecharny updated DIRSERVER-731:
----------------------------------------
Fix Version/s: 2.0.0-RC2
(was: 2.0.0-RC1)
Critical, but too vast to be implemented in RC1.
> Streaming big objects
> ---------------------
>
> Key: DIRSERVER-731
> URL: https://issues.apache.org/jira/browse/DIRSERVER-731
> Project: Directory ApacheDS
> Issue Type: Improvement
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0-RC2
>
>
> We really need to implement a streming of big objects, like JpegPhoto, otherwise the server can be killed trying to manage them.
> The problem is that they are stored entirely in memory, and this is not an option for megabytes of data in a JVM. We should find a way to work with MINA to send bytes on the fly (for instance by packets of 8kb). The very same when we receive huge data : we should store them somwhere on the disk instead of creating a byte[] large enough to store the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-731) Streaming big objects
Posted by "Alex Karasulu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12520713 ]
Alex Karasulu commented on DIRSERVER-731:
-----------------------------------------
I agree that this is the ultimate way to solve the issue however this might take considerable
time to implement. Perhaps the best strategy for the moment is to create a special configuration
parameter for the twix decoder that limits the size of a TLV length to some value. This would
help prevent potential DoS attacks that flood the server with massive attributes which will
eventually result in OutOfMemoryErrors. Using a configurable parameter
like a max TLV length limit parameter in theory could be easily implemented. I was looking
at the code to see how this would be done but I must admit I was just confused by the code
which has gotten more complex than I can handle. However the key to doing this may be in
the Asn1Decoder in the shared-asn1 (org.apache.directory.shared.asn1.ber package). If it
could take this parameter and limit the length of the TLV by throwing a special exception for
this case then we would be able to prevent this kind of DoS attack.
Emmanuel if you can assess just how much effort this workaround would take then we can
decide whether or not we should implement it and create a separate issue for it. Then we
can just move this issue to be thoroughly handled in a 2.0 or 3.0 release where we have the
time to cleanup and redesign the codec with these advanced features in mind.
How does this sound?
> Streaming big objects
> ---------------------
>
> Key: DIRSERVER-731
> URL: https://issues.apache.org/jira/browse/DIRSERVER-731
> Project: Directory ApacheDS
> Issue Type: Improvement
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 1.5.1
>
>
> We really need to implement a streming of big objects, like JpegPhoto, otherwise the server can be killed trying to manage them.
> The problem is that they are stored entirely in memory, and this is not an option for megabytes of data in a JVM. We should find a way to work with MINA to send bytes on the fly (for instance by packets of 8kb). The very same when we receive huge data : we should store them somwhere on the disk instead of creating a byte[] large enough to store the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DIRSERVER-731) Streaming big objects
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Lecharny updated DIRSERVER-731:
----------------------------------------
Fix Version/s: 1.5.1
I would like to try to implement this feature in 1.5.1... Not sure I will succeed ;)
> Streaming big objects
> ---------------------
>
> Key: DIRSERVER-731
> URL: https://issues.apache.org/jira/browse/DIRSERVER-731
> Project: Directory ApacheDS
> Issue Type: Improvement
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 1.5.1
>
>
> We really need to implement a streming of big objects, like JpegPhoto, otherwise the server can be killed trying to manage them.
> The problem is that they are stored entirely in memory, and this is not an option for megabytes of data in a JVM. We should find a way to work with MINA to send bytes on the fly (for instance by packets of 8kb). The very same when we receive huge data : we should store them somwhere on the disk instead of creating a byte[] large enough to store the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DIRSERVER-731) Streaming big objects
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Lecharny updated DIRSERVER-731:
----------------------------------------
Fix Version/s: 2.0.0-RC1
(was: 2.0.0)
Moved back to 2.0.0-RC1
> Streaming big objects
> ---------------------
>
> Key: DIRSERVER-731
> URL: https://issues.apache.org/jira/browse/DIRSERVER-731
> Project: Directory ApacheDS
> Issue Type: Improvement
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0-RC1
>
>
> We really need to implement a streming of big objects, like JpegPhoto, otherwise the server can be killed trying to manage them.
> The problem is that they are stored entirely in memory, and this is not an option for megabytes of data in a JVM. We should find a way to work with MINA to send bytes on the fly (for instance by packets of 8kb). The very same when we receive huge data : we should store them somwhere on the disk instead of creating a byte[] large enough to store the data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DIRSERVER-731) Streaming big objects
Posted by "Trustin Lee (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/DIRSERVER-731?page=all ]
Trustin Lee updated DIRSERVER-731:
----------------------------------
Summary: Streaming big objects (was: Streming big objects)
> Streaming big objects
> ---------------------
>
> Key: DIRSERVER-731
> URL: http://issues.apache.org/jira/browse/DIRSERVER-731
> Project: Directory ApacheDS
> Issue Type: Improvement
> Reporter: Emmanuel Lecharny
> Priority: Critical
>
> We really need to implement a streming of big objects, like JpegPhoto, otherwise the server can be killed trying to manage them.
> The problem is that they are stored entirely in memory, and this is not an option for megabytes of data in a JVM. We should find a way to work with MINA to send bytes on the fly (for instance by packets of 8kb). The very same when we receive huge data : we should store them somwhere on the disk instead of creating a byte[] large enough to store the data.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira