You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Noah Slater <ns...@apache.org> on 2009/09/20 15:06:55 UTC

Second call for objections releasing 0.10

Hey,

I've been following the first thread, but am unsure where we all stand. This is
my second call for objections following our previous discussion. Do we all feel
ready to prepare and vote on the 0.10 release now?

Thanks,

-- 
Noah Slater, http://tumbolia.org/nslater

Re: Second call for objections releasing 0.10

Posted by Curt Arnold <ca...@apache.org>.
On Sep 20, 2009, at 4:48 PM, Chris Anderson wrote:
>
> That's the sort of thing that'd get backported for 0.10.1 anyway, so I
> don't think it's a blocker. Also, probably a fairly easy patch.
>
> Chris
>


COUCHDB-345 seemed to get no attention in the last call for objections  
(other than the fix to a unit test that would break if a patch were  
applied).

I believe the problem addressed in the issue that makes every CouchDB  
installation that allows untrusted users to write to the database  
vulnerable.  Also, as far as I know there is not a simple procedure to  
recover a CouchDB that has been wedged by a malicious or unintentional  
insert of a malencoded document.

No one has objected to the badenc1.patch, however I believe the  
performance cost could be reduced by first scanning the incoming byte  
array and only calling xmerl_ucs:from_utf8 on the portion beginning  
with the first byte value >= 0x80.  I'm not confident in my Erlang  
skills yet to think that I know the optimal way of coding that.   
However, I think it would be better to get some fix in than wait for  
an optimal fix.



Re: Second call for objections releasing 0.10

Posted by Jan Lehnardt <ja...@apache.org>.
On 22 Sep 2009, at 06:50, Benoit Chesneau wrote:

> On Mon, Sep 21, 2009 at 6:50 PM, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> I think I see where this is coming from, but rather see it  
>> perceived that
>> bugfix updates to a stable branch should happen in a timely manner.  
>> We can't
>> hold back releases forever and we should demonstrate that we can  
>> release
>> 0.10.1 quickly, if needed.
>>
>
> I didn't say apache process is bad anyway. I agree with you on this
> point. And I'm perfectly ok with agile development as you know that ;)
>
> But in any case, I find it isn't normal to have spurious logs in
> production that may have others side effects. This is difficult to
> have such logs when you have a product online or in tests and I
> wouldn't use a service that accept such logs without knowing if it has
> side effects or not (and then logs should be more quiet anyway to
> allow read of others). I just opened the bug because I was testing
> that when noah called for the release, but of course it could have
> been detected before and maybe that's the point. In my opinion,
> something could be done to test most of expected scenarios before
> release. It won't be perfect but in this case, it would have help I
> guess.
>
>> I hope this addresses your concern.
>>
>
> Sure, sorry for my french. I didn't want to be rude. I was just
> worried that for once time couchdb could become unstable.

Heh, I didn't perceive your comment as rude. I was just worried that
we are stuck in a process that can be optimized. If you are confident
we can get bugfix releases out in a timely manner, I'm satisfied :)

Cheers
Jan
--


Re: Second call for objections releasing 0.10

Posted by Benoit Chesneau <bc...@gmail.com>.
On Mon, Sep 21, 2009 at 6:50 PM, Jan Lehnardt <ja...@apache.org> wrote:
>
> I think I see where this is coming from, but rather see it perceived that
> bugfix updates to a stable branch should happen in a timely manner. We can't
> hold back releases forever and we should demonstrate that we can release
> 0.10.1 quickly, if needed.
>

I didn't say apache process is bad anyway. I agree with you on this
point. And I'm perfectly ok with agile development as you know that ;)

But in any case, I find it isn't normal to have spurious logs in
production that may have others side effects. This is difficult to
have such logs when you have a product online or in tests and I
wouldn't use a service that accept such logs without knowing if it has
side effects or not (and then logs should be more quiet anyway to
allow read of others). I just opened the bug because I was testing
that when noah called for the release, but of course it could have
been detected before and maybe that's the point. In my opinion,
something could be done to test most of expected scenarios before
release. It won't be perfect but in this case, it would have help I
guess.

> I hope this addresses your concern.
>

Sure, sorry for my french. I didn't want to be rude. I was just
worried that for once time couchdb could become unstable.

- benoît

Re: Second call for objections releasing 0.10

Posted by Jan Lehnardt <ja...@apache.org>.
On 21.09.2009, at 06:08, Benoit Chesneau <bc...@gmail.com> wrote:

> On Sun, Sep 20, 2009 at 11:48 PM, Chris Anderson
>> That's the sort of thing that'd get backported for 0.10.1 anyway,  
>> so I
>> don't think it's a blocker. Also, probably a fairly easy patch.
>>
>> Chris
>>
> Well it's really annoying to have such error in logs when you are in
> production. I've also updated the bug : when you delete a db witch is
> continuously replicatedit create an error too on the machine handling
> replication when I delete a database on source. There maybe other side
> effect.
>
> More over, due to apache process, 10.1 could appear very long time
> after 10.0 and imo having this known obvious errors fixed before the
> release may be better.

I think I see where this is coming from, but rather see it perceived  
that bugfix updates to a stable branch should happen in a timely  
manner. We can't hold back releases forever and we should demonstrate  
that we can release 0.10.1 quickly, if needed.

I hope this addresses your concern.

Cheers
Jan
--
>

Re: Second call for objections releasing 0.10

Posted by Adam Kocoloski <ko...@apache.org>.
On Sep 21, 2009, at 1:08 AM, Benoit Chesneau wrote:

> On Sun, Sep 20, 2009 at 11:48 PM, Chris Anderson
>> That's the sort of thing that'd get backported for 0.10.1 anyway,  
>> so I
>> don't think it's a blocker. Also, probably a fairly easy patch.
>>
>> Chris
>>
> Well it's really annoying to have such error in logs when you are in
> production. I've also updated the bug : when you delete a db witch is
> continuously replicatedit create an error too on the machine handling
> replication when I delete a database on source. There maybe other side
> effect.

So, we fixed a couple of bugs affecting _changes when a DB is deleted  
mid-response, but it's not perfect.  In particular:

1) There's no indication to the end user that _changes terminated  
because the DB was deleted.  Benoit wrote up such a patch in  
COUCHDB-508, but I didn't want to alter the _changes format so close  
to the release.

2) The replicator may still puke out quite a bit of Erlang tracebacks  
if you delete a DB that is currently a replication source or target.   
I think some of it is unavoidable -- deleting a DB out from underneath  
a replication should be a rare occurrence, and for those rare  
occurrences it makes sense to use exit codes other than "normal" so  
that other related processes are also terminated.  When we do that,  
tracebacks are the immediate result.  With that said, we can still  
slim down what's in place now and avoid some duplication of error  
messages.  I don't think that goal should hold up 0.10.

Best, Adam

Re: Second call for objections releasing 0.10

Posted by Benoit Chesneau <bc...@gmail.com>.
On Sun, Sep 20, 2009 at 11:48 PM, Chris Anderson
> That's the sort of thing that'd get backported for 0.10.1 anyway, so I
> don't think it's a blocker. Also, probably a fairly easy patch.
>
> Chris
>
Well it's really annoying to have such error in logs when you are in
production. I've also updated the bug : when you delete a db witch is
continuously replicatedit create an error too on the machine handling
replication when I delete a database on source. There maybe other side
effect.

More over, due to apache process, 10.1 could appear very long time
after 10.0 and imo having this known obvious errors fixed before the
release may be better.


- benoît

Re: Second call for objections releasing 0.10

Posted by Chris Anderson <jc...@apache.org>.
On Sun, Sep 20, 2009 at 6:35 AM, Benoit Chesneau <bc...@gmail.com> wrote:
> On Sun, Sep 20, 2009 at 3:06 PM, Noah Slater <ns...@apache.org> wrote:
>> Hey,
>>
>> I've been following the first thread, but am unsure where we all stand. This is
>> my second call for objections following our previous discussion. Do we all feel
>> ready to prepare and vote on the 0.10 release now?
>>
>> Thanks,
>>
>> --
>> Noah Slater, http://tumbolia.org/nslater
>>
>
> depends on what kocolosk will say about it, but this error in 0.10
> replication could be a blocker :
>
> https://issues.apache.org/jira/browse/COUCHDB-506
>

That's the sort of thing that'd get backported for 0.10.1 anyway, so I
don't think it's a blocker. Also, probably a fairly easy patch.

Chris



-- 
Chris Anderson
http://jchrisa.net
http://couch.io

Re: Second call for objections releasing 0.10

Posted by Adam Kocoloski <ko...@apache.org>.
On Sep 20, 2009, at 9:35 AM, Benoit Chesneau wrote:

> On Sun, Sep 20, 2009 at 3:06 PM, Noah Slater <ns...@apache.org>  
> wrote:
>> Hey,
>>
>> I've been following the first thread, but am unsure where we all  
>> stand. This is
>> my second call for objections following our previous discussion. Do  
>> we all feel
>> ready to prepare and vote on the 0.10 release now?
>>
>> Thanks,
>>
>> --
>> Noah Slater, http://tumbolia.org/nslater
>>
>
> depends on what kocolosk will say about it, but this error in 0.10
> replication could be a blocker :
>
> https://issues.apache.org/jira/browse/COUCHDB-506

That issue is now closed, I think we can move forward.  Best,

Adam


Re: Second call for objections releasing 0.10

Posted by Benoit Chesneau <bc...@gmail.com>.
On Sun, Sep 20, 2009 at 3:06 PM, Noah Slater <ns...@apache.org> wrote:
> Hey,
>
> I've been following the first thread, but am unsure where we all stand. This is
> my second call for objections following our previous discussion. Do we all feel
> ready to prepare and vote on the 0.10 release now?
>
> Thanks,
>
> --
> Noah Slater, http://tumbolia.org/nslater
>

depends on what kocolosk will say about it, but this error in 0.10
replication could be a blocker :

https://issues.apache.org/jira/browse/COUCHDB-506


- benoît

Re: Second call for objections releasing 0.10

Posted by Paul Davis <pa...@gmail.com>.
On Mon, Sep 21, 2009 at 3:28 PM, Noah Slater <ns...@apache.org> wrote:
> On Mon, Sep 21, 2009 at 10:57:03AM -0400, Paul Davis wrote:
>> I updated all of the config files for the etap tests to include an ASF
>> license header. As per [1] it looks like we should do the same for
>> default.ini.tpl.in and local.ini.
>
> No, as previously discussed, I do not want to do this.
>
> Thanks,
>
> --
> Noah Slater, http://tumbolia.org/nslater
>

The only previous mention I see is [1]. When I read the license page I
was in doubt and defaulted to including them for the test configs. I
just had the thought of checking HTTPd's config [2] and it doesn't
include them so I'm convinced they're not considered creative.

And I return you to your normally scheduled programming.

[1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/200806.mbox/%3c20080617203510.GE1027@bytesexual.org%3e
[2] http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/conf/httpd.conf.in?view=markup

Re: Second call for objections releasing 0.10

Posted by Noah Slater <ns...@apache.org>.
On Mon, Sep 21, 2009 at 10:57:03AM -0400, Paul Davis wrote:
> I updated all of the config files for the etap tests to include an ASF
> license header. As per [1] it looks like we should do the same for
> default.ini.tpl.in and local.ini.

No, as previously discussed, I do not want to do this.

Thanks,

-- 
Noah Slater, http://tumbolia.org/nslater

Re: Second call for objections releasing 0.10

Posted by Paul Davis <pa...@gmail.com>.
On Mon, Sep 21, 2009 at 10:15 AM, Noah Slater <ns...@apache.org> wrote:
> On Sun, Sep 20, 2009 at 11:53:09PM -0500, Curt Arnold wrote:
>> test/etap/041-uuid-gen-seq.ini
>> test/etap/041-uuid-gen-utc.ini
>> test/etap/121-stats-aggregates.ini
>>
>> do not have license notices and are not listed in license.skip.  I'm
>> assuming that you can feed license.skip into RAT with some option that
>> I've overlooked.
>
> I re-implemented RAT in four lines of GNU Make:
>
>  distcheck-hook:
>          grep -rL 'http://www.apache.org/licenses/LICENSE-2.0' * \
>              | grep -vEf license.skip; \
>          test "$$?" -eq 1
>
> Needless to say, the tarball won't build if this fails.
>
>> I think it would be best to add an ASF license notice to all of those
>> files.
>
> I will have pick these up while trying to build the tarball.
>
> Thanks for the attention to detail.
>
> Best,
>
> --
> Noah Slater, http://tumbolia.org/nslater
>

I updated all of the config files for the etap tests to include an ASF
license header. As per [1] it looks like we should do the same for
default.ini.tpl.in and local.ini.

Paul

[1] http://www.apache.org/legal/src-headers.html

Re: Second call for objections releasing 0.10

Posted by Noah Slater <ns...@apache.org>.
On Sun, Sep 20, 2009 at 11:53:09PM -0500, Curt Arnold wrote:
> test/etap/041-uuid-gen-seq.ini
> test/etap/041-uuid-gen-utc.ini
> test/etap/121-stats-aggregates.ini
>
> do not have license notices and are not listed in license.skip.  I'm
> assuming that you can feed license.skip into RAT with some option that
> I've overlooked.

I re-implemented RAT in four lines of GNU Make:

  distcheck-hook:
          grep -rL 'http://www.apache.org/licenses/LICENSE-2.0' * \
              | grep -vEf license.skip; \
          test "$$?" -eq 1

Needless to say, the tarball won't build if this fails.

> I think it would be best to add an ASF license notice to all of those
> files.

I will have pick these up while trying to build the tarball.

Thanks for the attention to detail.

Best,

-- 
Noah Slater, http://tumbolia.org/nslater

Re: Second call for objections releasing 0.10

Posted by Curt Arnold <ca...@apache.org>.
On Sep 20, 2009, at 8:06 AM, Noah Slater wrote:

> Hey,
>
> I've been following the first thread, but am unsure where we all  
> stand. This is
> my second call for objections following our previous discussion. Do  
> we all feel
> ready to prepare and vote on the 0.10 release now?
>
> Thanks,
>
> -- 
> Noah Slater, http://tumbolia.org/nslater


test/etap/041-uuid-gen-seq.ini
test/etap/041-uuid-gen-utc.ini
test/etap/121-stats-aggregates.ini

do not have license notices and are not listed in license.skip.  I'm  
assuming that you can feed license.skip into RAT with some option that  
I've overlooked.

.gitignore does not have a license header, but I'm guessing it would  
not be included in the distribution.

I think it would be best to add an ASF license notice to all of those  
files.

Re: Second call for objections releasing 0.10

Posted by Paul Davis <pa...@gmail.com>.
On Mon, Sep 21, 2009 at 2:12 PM, Curt Arnold <cu...@gmail.com> wrote:
> Probably should not have ventured any opinion on a resolution since I had
> just scanned the thread and hadn't had any time to investigate it myself.
> However, it did seem to suggest a possible means of abuse and didn't seem to
> have a resolution at the moment.
>
> Doing a quick test with IE and FF, showed that:
>
> { withdrawl: 10000.00, withdrawl: 10.00 }
>
> appeared to be indistinguishable from:
>
> { withdrawl : 10.00 }
>
> Basically, it was as if you did:
>
> var doc = new Object();
> doc.withdrawl = 10000.0;
> doc.withdrawl = 10.0;
>
> and the first assignment got discarded.
>

I reckon that this would be the behavior of most parsers. I haven't
seen any that die on duplication, they generally just assign each key
to the language's object mapping type overwriting any previous value.
I think maybe I remember seeing Yajl have an option to call it an
error but I'd have to double check as I could just be making that up.

> As long as CouchDB is consistent with that interpretation, then there is not
> a problem as far as I can tell.
>
> If some level (like the query processor) saw both entries, then you might be
> able to exploit the discrepancy.
>

There's probably a possibility in confusing tools that don't expect to
see repeated JSON properties, but that won't get fixed by us rejecting
documents. We could just as well react like other tools and only
accept the final definition of the property, but who's to say that's
proper? As I remember it, there's no resolution for duplicate names
mentioned in the spec. And its not specified as an error condition.

My summary of this would be that its a fairly uncommon bug. I've seen
this one report in the two years or so I've been working with CouchDB.
To trigger the behavior you're pretty much required to write your own
JSON serializer as most standard implementations won't allow it. As to
a malicious attack, I'm not sure I see it. The only place people would
even notice it would be in the Erlang view code. And at that point I
would think that it'd just be a documentation issue when people are
doing security.

Even thinking forward to consider JSON serialization BIF's I doubt
that they would enforce unique keys by default as the preferred Erlang
representation uses a data structure without uniqueness constraints.
It'd be possible that there's an included option to check, but I don't
remember anything in the EEP about that.

Bottom line, I think we should just Relax on this for now. Unless
someone can illustrate even the remote possibility of an attack that
this would provide, or some sort of extreme incompatibility like
causing IE5.5 to segfault I don't consider it something that should be
worried about.

Also, once again to reiterate, this isn't *invalid* JSON. Just JSON
that people might not expect. And its only there if a client provided
it. CouchDB never generates duplicate properties to my knowledge.

HTH,
Paul Davis

Re: Second call for objections releasing 0.10

Posted by Curt Arnold <cu...@gmail.com>.
Probably should not have ventured any opinion on a resolution since I had
just scanned the thread and hadn't had any time to investigate it myself.
However, it did seem to suggest a possible means of abuse and didn't seem to
have a resolution at the moment.

Doing a quick test with IE and FF, showed that:

{ withdrawl: 10000.00, withdrawl: 10.00 }

appeared to be indistinguishable from:

{ withdrawl : 10.00 }

Basically, it was as if you did:

var doc = new Object();
doc.withdrawl = 10000.0;
doc.withdrawl = 10.0;

and the first assignment got discarded.

As long as CouchDB is consistent with that interpretation, then there is not
a problem as far as I can tell.

If some level (like the query processor) saw both entries, then you might be
able to exploit the discrepancy.

Re: Second call for objections releasing 0.10

Posted by Paul Davis <pa...@gmail.com>.
On Mon, Sep 21, 2009 at 10:08 AM, Curt Arnold <ca...@apache.org> wrote:
>
> On Sep 20, 2009, at 8:06 AM, Noah Slater wrote:
>
>> Hey,
>>
>> I've been following the first thread, but am unsure where we all stand.
>> This is
>> my second call for objections following our previous discussion. Do we all
>> feel
>> ready to prepare and vote on the 0.10 release now?
>>
>> Thanks,
>>
>> --
>> Noah Slater, http://tumbolia.org/nslater
>
>
> The "invalid json allowed into CouchDB" thread on
> user@couchdb.apache.org appears to offer a means of gaming a system to place
> data in a document that would be seen by CouchDB, but could be hidden from
> clients.
>
> I think the issue needs to be resolved before cutting 0.10.  It does appear
> to be a security issue, but one which the resolution could negatively impact
> some fraction of apps that depended on the behavior.  If we address it
> before 0.10, we could just reject the document as invalid.  As a security
> patch in a 0.10.1 or so, we may feel compelled to try to merge the data to
> preserve the rare app that depended on the function.
>
> My initial opinion is that any document with multiple occurrences of a
> property should be rejected and it could just be weaved into the patch for
> COUCHDB-345.
>

As I described in the user@ thread, the status of repeated property
names in JSON is arguable in either direction. I find it a bit of a
stretch to label this a security concern though. I do agree that it
could present some irritating behavior in places, and it theoretically
might be possible to break something somewhere but I'm having a hard
time thinking of something specific to CouchDB that would break.

Attempting to merge fields in JSON automatically is completely out of
the question though. We either allow repeated fields or we don't.
Either behavior is acceptable to me with a couple notes:

Checking for repeated properties is gonna be a bit of a performance
hit in the JSON parser. The Erlang parsers all allow repeated
properties because the underlying structure is a proplist, which is
just a list of key-value tuples. Checking for repeats would either be
a call to one of the proplist normalization functions and checking for
identity or a sort and foldl to check. No idea how much of a hit
that'd be exactly, but I imagine it could be measured for anything
bigger than trivial docs.

Secondly, if we do add this check, we'll have to consider what happens
with databases that already have docs with repeated fields. The
simplest way would be to just keep serving content and reject new
edits. Which is kind of weird. But forcing a dump/reload for *all*
databases for what I would think is a relatively small number of
affected db's seems cumbersome. Perhaps a tool that would dump a db
and determine if it was in offense would be all that's needed.

HTH,
Paul Davis

Not to future self:

lists:foldl(fun({K, V}, P) when P =/= K -> K end, lists:sort(Props),
not_a_json_key)

That popped into my head. Figured I should write it down before it un pops.

Re: Second call for objections releasing 0.10

Posted by Curt Arnold <ca...@apache.org>.
On Sep 20, 2009, at 8:06 AM, Noah Slater wrote:

> Hey,
>
> I've been following the first thread, but am unsure where we all  
> stand. This is
> my second call for objections following our previous discussion. Do  
> we all feel
> ready to prepare and vote on the 0.10 release now?
>
> Thanks,
>
> -- 
> Noah Slater, http://tumbolia.org/nslater


The "invalid json allowed into CouchDB" thread on user@couchdb.apache.org 
  appears to offer a means of gaming a system to place data in a  
document that would be seen by CouchDB, but could be hidden from  
clients.

I think the issue needs to be resolved before cutting 0.10.  It does  
appear to be a security issue, but one which the resolution could  
negatively impact some fraction of apps that depended on the  
behavior.  If we address it before 0.10, we could just reject the  
document as invalid.  As a security patch in a 0.10.1 or so, we may  
feel compelled to try to merge the data to preserve the rare app that  
depended on the function.

My initial opinion is that any document with multiple occurrences of a  
property should be rejected and it could just be weaved into the patch  
for COUCHDB-345.