You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by Jeff Young <je...@adobe.com> on 2011/12/01 22:26:21 UTC
FW: Issue with DOS limitation in infinity.json servlet
The intent behind the limitation seems sound, but the implementation has (to my mind) a slight flaw.
A legitimate client which needs the information could presumably implement its own traversal to descend the tree. But this only works if the json servlet is always allowed to return at least a depth of 1. The current implementation limits the depth to 0 if the node in question has more than the limit number of children.
I was discussing this with Alex, who pointed out that the intent was to be defensive. However, if we really want to limit the *number of children* a node can have, then we ought to do that elsewhere. Given that a node *does* have a certain number of children, the json servlet needs to at least support the enumeration of said children.
So I'd like to propose that we amend the DOS-protection-algorithm to stop at 1, rather than 0.
Thoughts?
Thanks,
Jeff.
(PS: apologies if this gets sent out twice, but I think ezmlm ate the first posting because I hadn't yet cofirmed my subscription so I'm re-sending.)
Jeff Young | Principal Scientist | Adobe Distinguished Inventor
Adobe Systems Software Ireland Ltd.
Registered Office: 4-6 Riverwalk, Citywest Business Campus,
Saggart, Dublin 24, Ireland Company No. 344992
P Please consider your environmental responsibility before printing this e-mail.
Re: Issue with DOS limitation in infinity.json servlet
Posted by Carsten Ziegeler <cz...@apache.org>.
+1
Carsten
2011/12/2 Bertrand Delacretaz <bd...@apache.org>:
> On Fri, Dec 2, 2011 at 1:00 PM, Felix Meschberger <fm...@adobe.com> wrote:
>> ....I am inclined to agree, that the first level of child nodes to return should count into the overall limitation counter but should not be limited by that counter.
>>
>> Thus the first level of child nodes is always returned (if only as "pointers" or entries) while starting with the second level, the limit is enforced....
>
> I agree with that, sounds like the least surprise in this case.
>
>>
>> Patch welcome, I might say ;-)
>
> I also agree with that, also sounds like the least surprise in this case ;-)
>
> -Bertrand
--
Carsten Ziegeler
cziegeler@apache.org
Re: Issue with DOS limitation in infinity.json servlet
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Fri, Dec 2, 2011 at 1:00 PM, Felix Meschberger <fm...@adobe.com> wrote:
> ....I am inclined to agree, that the first level of child nodes to return should count into the overall limitation counter but should not be limited by that counter.
>
> Thus the first level of child nodes is always returned (if only as "pointers" or entries) while starting with the second level, the limit is enforced....
I agree with that, sounds like the least surprise in this case.
>
> Patch welcome, I might say ;-)
I also agree with that, also sounds like the least surprise in this case ;-)
-Bertrand
Re: Issue with DOS limitation in infinity.json servlet
Posted by Felix Meschberger <fm...@adobe.com>.
Hi,
I am inclined to agree, that the first level of child nodes to return should count into the overall limitation counter but should not be limited by that counter.
Thus the first level of child nodes is always returned (if only as "pointers" or entries) while starting with the second level, the limit is enforced.
Patch welcome, I might say ;-)
Regards
Felix
Am 01.12.2011 um 22:26 schrieb Jeff Young:
> The intent behind the limitation seems sound, but the implementation has (to my mind) a slight flaw.
>
> A legitimate client which needs the information could presumably implement its own traversal to descend the tree. But this only works if the json servlet is always allowed to return at least a depth of 1. The current implementation limits the depth to 0 if the node in question has more than the limit number of children.
>
> I was discussing this with Alex, who pointed out that the intent was to be defensive. However, if we really want to limit the *number of children* a node can have, then we ought to do that elsewhere. Given that a node *does* have a certain number of children, the json servlet needs to at least support the enumeration of said children.
>
> So I'd like to propose that we amend the DOS-protection-algorithm to stop at 1, rather than 0.
>
> Thoughts?
>
> Thanks,
> Jeff.
>
> (PS: apologies if this gets sent out twice, but I think ezmlm ate the first posting because I hadn't yet cofirmed my subscription so I'm re-sending.)
>
>
>
> Jeff Young | Principal Scientist | Adobe Distinguished Inventor
> Adobe Systems Software Ireland Ltd.
> Registered Office: 4-6 Riverwalk, Citywest Business Campus,
> Saggart, Dublin 24, Ireland Company No. 344992
> P Please consider your environmental responsibility before printing this e-mail.
>
>
RE: FW: Issue with DOS limitation in infinity.json servlet
Posted by Jeff Young <je...@adobe.com>.
Hi Justin,
Yes, the customer could increase the limit, but realistically only after a support call.
But I'm more concerned about the component developer who wants to avoid that support call. Their logical next step is to write their own single-level JSON JSP, and that seems like the platform letting them down.
Jeff.
-----Original Message-----
From: justinedelson@gmail.com [mailto:justinedelson@gmail.com] On Behalf Of Justin Edelson
Sent: 01 December 2011 21:50
To: dev@sling.apache.org
Subject: Re: FW: Issue with DOS limitation in infinity.json servlet
Hi Jeff,
I'm not sure why you can't just increase the limit if you run into
this problem, but I am not opposed to making this change on principal.
I'm very intrigued by the idea of a PostProcessor which limits the
number of nodes at a particular point in the hierarchy, but that's not
going to be 100% effective as Sling doesn't "own" the repository per
se.
Justin
On Thu, Dec 1, 2011 at 4:26 PM, Jeff Young <je...@adobe.com> wrote:
> The intent behind the limitation seems sound, but the implementation has (to my mind) a slight flaw.
>
> A legitimate client which needs the information could presumably implement its own traversal to descend the tree. But this only works if the json servlet is always allowed to return at least a depth of 1. The current implementation limits the depth to 0 if the node in question has more than the limit number of children.
>
> I was discussing this with Alex, who pointed out that the intent was to be defensive. However, if we really want to limit the *number of children* a node can have, then we ought to do that elsewhere. Given that a node *does* have a certain number of children, the json servlet needs to at least support the enumeration of said children.
>
> So I'd like to propose that we amend the DOS-protection-algorithm to stop at 1, rather than 0.
>
> Thoughts?
>
> Thanks,
> Jeff.
>
> (PS: apologies if this gets sent out twice, but I think ezmlm ate the first posting because I hadn't yet cofirmed my subscription so I'm re-sending.)
>
>
>
> Jeff Young | Principal Scientist | Adobe Distinguished Inventor
> Adobe Systems Software Ireland Ltd.
> Registered Office: 4-6 Riverwalk, Citywest Business Campus,
> Saggart, Dublin 24, Ireland Company No. 344992
> P Please consider your environmental responsibility before printing this e-mail.
>
>
RE: FW: Issue with DOS limitation in infinity.json servlet
Posted by Jeff Young <je...@adobe.com>.
I think it's this "know in advance that a tree is going to be big" concept
that we're currently running afoul of.
Let's say that I pretty much "know" that my tree will be small. So I call
x/y/z.infinity.json. All well and good.
And in another case, that I "know" that my tree is likely to be colossal,
so I implement a search rather than an enumeration. Good again.
But in a third case, I "know" that my tree might be "large-ish", so I
implement a lazy walk in steps as needed (say, for instance, through a tree
control). But if I can't make a single step, then I'm sunk.
One could argue that "large-ish" and "DOS" aren't really of the same
magnitude, and that we just need to raise the default limit. In the tree
control instance, even with type-ahead searching it's hard to imagine it
being useful for 10,000 entries. But it's clearly useful for 1000 entries
(I've got a customer to "prove" it ;) -- so maybe we just need to raise the
default to something like 8K (it's currently 1K).
But I still think limiting to one level is a cleaner solution. If a single
level has enough nodes to produce a DOS attack, then it seems like you've
got more worries than x/y/z.1.json.
Jeff.
-----Original Message-----
From: ianboston@gmail.com [mailto:ianboston@gmail.com] On Behalf Of Ian Boston
Sent: 02 December 2011 00:14
To: dev@sling.apache.org
Subject: Re: FW: Issue with DOS limitation in infinity.json servlet
On 2 December 2011 10:51, Justin Edelson <ju...@justinedelson.com> wrote:
> Hmmm. Good point. I changed my mind (slightly) - the behavior Jeff is
> describing can be supported, but should be disabled by default.
>
> Ian - to your question, with the default configuration,
> /x/y/z.(anything).json should not output 2M child nodes. IMHO, if you
> as a system operator decide to let clients request all 2M nodes, that
> should be your prerogative to do so via configuration.
Yes, agreed, if you know in advance that a tree is going to be big,
then it should be possible to configure it not to do something nasty
when a x.-1.json comes in. IIRC Sling already has this capability with
ResourceProvider and Resource.listChildren which can be implemented to
return no results.
Perhaps the protection against DOS should be removed, and users of
Sling should be encouraged to use the core Sling API to do the right
thing, just a thought ?
I think the implementation originally came from one of my use cases,
were we were trying to store "billions" of items in a multi level
tree.... we stopped doing that for all sorts of other reasons and used
the Resource.listChildren approach.
>
> In reality, however, if you have that type of structure, I think
> you're more likely to do content discovery by search rather than
> browse. WDYT?
yes very true,
Its interesting/depressing to see how often lists are used as a
substitute for what is really search. The feature request shortly
follows for sorting, and selecting by first letter, and if you watch a
user they use the paging, selecting, sorting in real frustration, when
all they wanted was to be able to search for the item slap bang in the
middle of the result set.
Ian
>
> Justin
>
> On Thu, Dec 1, 2011 at 5:53 PM, Ian Boston <ie...@tfd.co.uk> wrote:
>> Hi,
>> Thinking about this some more,
>> Assuming the content system can support this for a moment.
>> /x/y/z has 2M direct child nodes, what does /x/y/z.-1.json respond
>> with? 2M links to those child nodes.
>>
>> Does the system need to support paging, in the same way atom does?
>> eg /x/y/z.-1.json?page=1
>>
>> With Sling built on Jackrabbit this doesn't happen (yet), since
>> Jackrabbit (IIUC) stores child nodes internally as an array of
>> pointers on the parent node, so it may not be a real issue, but Sling
>> on something else may encounter this. In general the solution has been
>> to refuse to list child Resources via the Resource interface, and do
>> something custom with paging enforced, but as soon as paging is
>> introduced, order becomes relevant, and that opens up the validity of
>> ordering a map in json, which IIRC is defined as a bag not a list.
>>
>> Ian
>>
>> On 2 December 2011 08:50, Justin Edelson <ju...@justinedelson.com> wrote:
>>> Hi Jeff,
>>> I'm not sure why you can't just increase the limit if you run into
>>> this problem, but I am not opposed to making this change on principal.
>>>
>>> I'm very intrigued by the idea of a PostProcessor which limits the
>>> number of nodes at a particular point in the hierarchy, but that's not
>>> going to be 100% effective as Sling doesn't "own" the repository per
>>> se.
>>>
>>> Justin
>>>
>>> On Thu, Dec 1, 2011 at 4:26 PM, Jeff Young <je...@adobe.com> wrote:
>>>> The intent behind the limitation seems sound, but the implementation has (to my mind) a slight flaw.
>>>>
>>>> A legitimate client which needs the information could presumably implement its own traversal to descend the tree. But this only works if the json servlet is always allowed to return at least a depth of 1. The current implementation limits the depth to 0 if the node in question has more than the limit number of children.
>>>>
>>>> I was discussing this with Alex, who pointed out that the intent was to be defensive. However, if we really want to limit the *number of children* a node can have, then we ought to do that elsewhere. Given that a node *does* have a certain number of children, the json servlet needs to at least support the enumeration of said children.
>>>>
>>>> So I'd like to propose that we amend the DOS-protection-algorithm to stop at 1, rather than 0.
>>>>
>>>> Thoughts?
>>>>
>>>> Thanks,
>>>> Jeff.
>>>>
>>>> (PS: apologies if this gets sent out twice, but I think ezmlm ate the first posting because I hadn't yet cofirmed my subscription so I'm re-sending.)
>>>>
>>>>
>>>>
>>>> Jeff Young | Principal Scientist | Adobe Distinguished Inventor
>>>> Adobe Systems Software Ireland Ltd.
>>>> Registered Office: 4-6 Riverwalk, Citywest Business Campus,
>>>> Saggart, Dublin 24, Ireland Company No. 344992
>>>> P Please consider your environmental responsibility before printing this e-mail.
>>>>
>>>>
>>>
>
Re: FW: Issue with DOS limitation in infinity.json servlet
Posted by Ian Boston <ie...@tfd.co.uk>.
On 2 December 2011 10:51, Justin Edelson <ju...@justinedelson.com> wrote:
> Hmmm. Good point. I changed my mind (slightly) - the behavior Jeff is
> describing can be supported, but should be disabled by default.
>
> Ian - to your question, with the default configuration,
> /x/y/z.(anything).json should not output 2M child nodes. IMHO, if you
> as a system operator decide to let clients request all 2M nodes, that
> should be your prerogative to do so via configuration.
Yes, agreed, if you know in advance that a tree is going to be big,
then it should be possible to configure it not to do something nasty
when a x.-1.json comes in. IIRC Sling already has this capability with
ResourceProvider and Resource.listChildren which can be implemented to
return no results.
Perhaps the protection against DOS should be removed, and users of
Sling should be encouraged to use the core Sling API to do the right
thing, just a thought ?
I think the implementation originally came from one of my use cases,
were we were trying to store "billions" of items in a multi level
tree.... we stopped doing that for all sorts of other reasons and used
the Resource.listChildren approach.
>
> In reality, however, if you have that type of structure, I think
> you're more likely to do content discovery by search rather than
> browse. WDYT?
yes very true,
Its interesting/depressing to see how often lists are used as a
substitute for what is really search. The feature request shortly
follows for sorting, and selecting by first letter, and if you watch a
user they use the paging, selecting, sorting in real frustration, when
all they wanted was to be able to search for the item slap bang in the
middle of the result set.
Ian
>
> Justin
>
> On Thu, Dec 1, 2011 at 5:53 PM, Ian Boston <ie...@tfd.co.uk> wrote:
>> Hi,
>> Thinking about this some more,
>> Assuming the content system can support this for a moment.
>> /x/y/z has 2M direct child nodes, what does /x/y/z.-1.json respond
>> with? 2M links to those child nodes.
>>
>> Does the system need to support paging, in the same way atom does?
>> eg /x/y/z.-1.json?page=1
>>
>> With Sling built on Jackrabbit this doesn't happen (yet), since
>> Jackrabbit (IIUC) stores child nodes internally as an array of
>> pointers on the parent node, so it may not be a real issue, but Sling
>> on something else may encounter this. In general the solution has been
>> to refuse to list child Resources via the Resource interface, and do
>> something custom with paging enforced, but as soon as paging is
>> introduced, order becomes relevant, and that opens up the validity of
>> ordering a map in json, which IIRC is defined as a bag not a list.
>>
>> Ian
>>
>> On 2 December 2011 08:50, Justin Edelson <ju...@justinedelson.com> wrote:
>>> Hi Jeff,
>>> I'm not sure why you can't just increase the limit if you run into
>>> this problem, but I am not opposed to making this change on principal.
>>>
>>> I'm very intrigued by the idea of a PostProcessor which limits the
>>> number of nodes at a particular point in the hierarchy, but that's not
>>> going to be 100% effective as Sling doesn't "own" the repository per
>>> se.
>>>
>>> Justin
>>>
>>> On Thu, Dec 1, 2011 at 4:26 PM, Jeff Young <je...@adobe.com> wrote:
>>>> The intent behind the limitation seems sound, but the implementation has (to my mind) a slight flaw.
>>>>
>>>> A legitimate client which needs the information could presumably implement its own traversal to descend the tree. But this only works if the json servlet is always allowed to return at least a depth of 1. The current implementation limits the depth to 0 if the node in question has more than the limit number of children.
>>>>
>>>> I was discussing this with Alex, who pointed out that the intent was to be defensive. However, if we really want to limit the *number of children* a node can have, then we ought to do that elsewhere. Given that a node *does* have a certain number of children, the json servlet needs to at least support the enumeration of said children.
>>>>
>>>> So I'd like to propose that we amend the DOS-protection-algorithm to stop at 1, rather than 0.
>>>>
>>>> Thoughts?
>>>>
>>>> Thanks,
>>>> Jeff.
>>>>
>>>> (PS: apologies if this gets sent out twice, but I think ezmlm ate the first posting because I hadn't yet cofirmed my subscription so I'm re-sending.)
>>>>
>>>>
>>>>
>>>> Jeff Young | Principal Scientist | Adobe Distinguished Inventor
>>>> Adobe Systems Software Ireland Ltd.
>>>> Registered Office: 4-6 Riverwalk, Citywest Business Campus,
>>>> Saggart, Dublin 24, Ireland Company No. 344992
>>>> P Please consider your environmental responsibility before printing this e-mail.
>>>>
>>>>
>>>
>
Re: FW: Issue with DOS limitation in infinity.json servlet
Posted by Justin Edelson <ju...@justinedelson.com>.
Hmmm. Good point. I changed my mind (slightly) - the behavior Jeff is
describing can be supported, but should be disabled by default.
Ian - to your question, with the default configuration,
/x/y/z.(anything).json should not output 2M child nodes. IMHO, if you
as a system operator decide to let clients request all 2M nodes, that
should be your prerogative to do so via configuration.
In reality, however, if you have that type of structure, I think
you're more likely to do content discovery by search rather than
browse. WDYT?
Justin
On Thu, Dec 1, 2011 at 5:53 PM, Ian Boston <ie...@tfd.co.uk> wrote:
> Hi,
> Thinking about this some more,
> Assuming the content system can support this for a moment.
> /x/y/z has 2M direct child nodes, what does /x/y/z.-1.json respond
> with? 2M links to those child nodes.
>
> Does the system need to support paging, in the same way atom does?
> eg /x/y/z.-1.json?page=1
>
> With Sling built on Jackrabbit this doesn't happen (yet), since
> Jackrabbit (IIUC) stores child nodes internally as an array of
> pointers on the parent node, so it may not be a real issue, but Sling
> on something else may encounter this. In general the solution has been
> to refuse to list child Resources via the Resource interface, and do
> something custom with paging enforced, but as soon as paging is
> introduced, order becomes relevant, and that opens up the validity of
> ordering a map in json, which IIRC is defined as a bag not a list.
>
> Ian
>
> On 2 December 2011 08:50, Justin Edelson <ju...@justinedelson.com> wrote:
>> Hi Jeff,
>> I'm not sure why you can't just increase the limit if you run into
>> this problem, but I am not opposed to making this change on principal.
>>
>> I'm very intrigued by the idea of a PostProcessor which limits the
>> number of nodes at a particular point in the hierarchy, but that's not
>> going to be 100% effective as Sling doesn't "own" the repository per
>> se.
>>
>> Justin
>>
>> On Thu, Dec 1, 2011 at 4:26 PM, Jeff Young <je...@adobe.com> wrote:
>>> The intent behind the limitation seems sound, but the implementation has (to my mind) a slight flaw.
>>>
>>> A legitimate client which needs the information could presumably implement its own traversal to descend the tree. But this only works if the json servlet is always allowed to return at least a depth of 1. The current implementation limits the depth to 0 if the node in question has more than the limit number of children.
>>>
>>> I was discussing this with Alex, who pointed out that the intent was to be defensive. However, if we really want to limit the *number of children* a node can have, then we ought to do that elsewhere. Given that a node *does* have a certain number of children, the json servlet needs to at least support the enumeration of said children.
>>>
>>> So I'd like to propose that we amend the DOS-protection-algorithm to stop at 1, rather than 0.
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> Jeff.
>>>
>>> (PS: apologies if this gets sent out twice, but I think ezmlm ate the first posting because I hadn't yet cofirmed my subscription so I'm re-sending.)
>>>
>>>
>>>
>>> Jeff Young | Principal Scientist | Adobe Distinguished Inventor
>>> Adobe Systems Software Ireland Ltd.
>>> Registered Office: 4-6 Riverwalk, Citywest Business Campus,
>>> Saggart, Dublin 24, Ireland Company No. 344992
>>> P Please consider your environmental responsibility before printing this e-mail.
>>>
>>>
>>
Re: FW: Issue with DOS limitation in infinity.json servlet
Posted by Ian Boston <ie...@tfd.co.uk>.
Hi,
Thinking about this some more,
Assuming the content system can support this for a moment.
/x/y/z has 2M direct child nodes, what does /x/y/z.-1.json respond
with? 2M links to those child nodes.
Does the system need to support paging, in the same way atom does?
eg /x/y/z.-1.json?page=1
With Sling built on Jackrabbit this doesn't happen (yet), since
Jackrabbit (IIUC) stores child nodes internally as an array of
pointers on the parent node, so it may not be a real issue, but Sling
on something else may encounter this. In general the solution has been
to refuse to list child Resources via the Resource interface, and do
something custom with paging enforced, but as soon as paging is
introduced, order becomes relevant, and that opens up the validity of
ordering a map in json, which IIRC is defined as a bag not a list.
Ian
On 2 December 2011 08:50, Justin Edelson <ju...@justinedelson.com> wrote:
> Hi Jeff,
> I'm not sure why you can't just increase the limit if you run into
> this problem, but I am not opposed to making this change on principal.
>
> I'm very intrigued by the idea of a PostProcessor which limits the
> number of nodes at a particular point in the hierarchy, but that's not
> going to be 100% effective as Sling doesn't "own" the repository per
> se.
>
> Justin
>
> On Thu, Dec 1, 2011 at 4:26 PM, Jeff Young <je...@adobe.com> wrote:
>> The intent behind the limitation seems sound, but the implementation has (to my mind) a slight flaw.
>>
>> A legitimate client which needs the information could presumably implement its own traversal to descend the tree. But this only works if the json servlet is always allowed to return at least a depth of 1. The current implementation limits the depth to 0 if the node in question has more than the limit number of children.
>>
>> I was discussing this with Alex, who pointed out that the intent was to be defensive. However, if we really want to limit the *number of children* a node can have, then we ought to do that elsewhere. Given that a node *does* have a certain number of children, the json servlet needs to at least support the enumeration of said children.
>>
>> So I'd like to propose that we amend the DOS-protection-algorithm to stop at 1, rather than 0.
>>
>> Thoughts?
>>
>> Thanks,
>> Jeff.
>>
>> (PS: apologies if this gets sent out twice, but I think ezmlm ate the first posting because I hadn't yet cofirmed my subscription so I'm re-sending.)
>>
>>
>>
>> Jeff Young | Principal Scientist | Adobe Distinguished Inventor
>> Adobe Systems Software Ireland Ltd.
>> Registered Office: 4-6 Riverwalk, Citywest Business Campus,
>> Saggart, Dublin 24, Ireland Company No. 344992
>> P Please consider your environmental responsibility before printing this e-mail.
>>
>>
>
Re: FW: Issue with DOS limitation in infinity.json servlet
Posted by Justin Edelson <ju...@justinedelson.com>.
Hi Jeff,
I'm not sure why you can't just increase the limit if you run into
this problem, but I am not opposed to making this change on principal.
I'm very intrigued by the idea of a PostProcessor which limits the
number of nodes at a particular point in the hierarchy, but that's not
going to be 100% effective as Sling doesn't "own" the repository per
se.
Justin
On Thu, Dec 1, 2011 at 4:26 PM, Jeff Young <je...@adobe.com> wrote:
> The intent behind the limitation seems sound, but the implementation has (to my mind) a slight flaw.
>
> A legitimate client which needs the information could presumably implement its own traversal to descend the tree. But this only works if the json servlet is always allowed to return at least a depth of 1. The current implementation limits the depth to 0 if the node in question has more than the limit number of children.
>
> I was discussing this with Alex, who pointed out that the intent was to be defensive. However, if we really want to limit the *number of children* a node can have, then we ought to do that elsewhere. Given that a node *does* have a certain number of children, the json servlet needs to at least support the enumeration of said children.
>
> So I'd like to propose that we amend the DOS-protection-algorithm to stop at 1, rather than 0.
>
> Thoughts?
>
> Thanks,
> Jeff.
>
> (PS: apologies if this gets sent out twice, but I think ezmlm ate the first posting because I hadn't yet cofirmed my subscription so I'm re-sending.)
>
>
>
> Jeff Young | Principal Scientist | Adobe Distinguished Inventor
> Adobe Systems Software Ireland Ltd.
> Registered Office: 4-6 Riverwalk, Citywest Business Campus,
> Saggart, Dublin 24, Ireland Company No. 344992
> P Please consider your environmental responsibility before printing this e-mail.
>
>