[GitHub] [couchdb] davisp opened pull request #1574: Fix builtin _sum reduce function

["davisp (GitHub)" <gi...@apache.org>]

The builting _sum reduce function has no protection against overflowing
reduce values. Users can emit objects with enough unique keys to cause
the builtin _sum to create objects that are exceedingly large in the
inner nodes of the view B+Tree.

This change adds the same logic that applies to JavaScript reduce
functions to check if a reduce function is properly reducing its input.

## Testing recommendations

make check

## Checklist

- [x] Code is written and works correctly;
- [ ] Changes are covered by tests;
- [ ] Documentation reflects the changes;


[ Full content available at: https://github.com/apache/couchdb/pull/1574 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [couchdb] davisp commented on issue #1574: Fix builtin _sum reduce function

["davisp (GitHub)" <gi...@apache.org>]

Trivially reproduced with this Python script:

I wanted to get the PR out but I'm also going to try and contemplate how to test this a bit more directly.

```python
#!/usr/bin/env python

import base64
import json
import os
import random

import requests

S = requests.session()
S.auth = ("adm", "pwd")
S.headers["Content-Type"] = "application/json"
BASE = "http://127.0.0.1:15986/"
DB = BASE + "foo"
DDOC = DB + "/_design/bar"
VIEW = DDOC + "/_view/bam"
SIG = "1d2f27b7abd18837599f078927c7bc1f"


def gen_key():
    data = os.urandom(50)
    return base64.b64encode(data)


def gen_device():
    return {
        "tx_bytes": random.randint(1, 8192),
        "rx_bytes": random.randint(1, 8192)
    }


def gen_data():
    ret = {}
    for i in range(1024):
        ret[gen_key()] = gen_device()
    return ret


def load_docs():
    for i in range(25):
        print "Generating: %d - %d" % (i*100, (i+1)*100)
        docs = []
        for j in range(100):
            docs.append({
                "_id": "%06d" % ((i*100) + j),
                "data": gen_data()
            })
        body = json.dumps({"docs": docs})
        r = S.post(DB + "/_bulk_docs", data=body)
        r.raise_for_status()


def load_ddoc():
    ddoc = {
        "language": "javascript",
        "views": {
            "bam": {
                "map": """
                    function(doc) {
                        emit(doc._id, doc.data);
                    }""",
                #"reduce": "_sum"
                "reduce": """
                    function(keys, values) {
                        var obj = {};
                        for(var v in values) {
                            for(var k in values[v]) {
                                log(k);
                                obj[k] = values[v][k];
                            }
                        }
                        return obj;
                    }
                """
            }
        }
    }
    body = json.dumps(ddoc)
    r = S.put(DB + "/_design/bar", data=body)
    r.raise_for_status()


def get_size():
    fname = "dev/lib/node1/data/.foo_design/mrview/%s.view" % SIG
    return os.stat(fname).st_size


def get_task():
    r = S.get(BASE + "/_active_tasks")
    for t in r.json():
        return t

def main():
    S.delete(DB)
    r = S.put(DB)
    r.raise_for_status()
    load_docs()
    load_ddoc()

    r = S.get(VIEW, params={"limit": 0})
    r.raise_for_status()

    print "Initial build: %d" % get_size()

    r = S.post(DDOC + "/_compact")
    r.raise_for_status()

    print get_task()


if __name__ == "__main__":
    main()
```

[ Full content available at: https://github.com/apache/couchdb/pull/1574 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [couchdb] davisp closed pull request #1574: Fix builtin _sum reduce function

["davisp (GitHub)" <gi...@apache.org>]

[ pull request closed by davisp ]

[ Full content available at: https://github.com/apache/couchdb/pull/1574 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [couchdb] davisp commented on pull request #1574: Fix builtin _sum reduce function

["davisp (GitHub)" <gi...@apache.org>]

Ah, good catch. I refactored that out intending to have it both places.

[ Full content available at: https://github.com/apache/couchdb/pull/1574 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [couchdb] davisp commented on issue #1574: Fix builtin _sum reduce function

["davisp (GitHub)" <gi...@apache.org>]

@nickva pointed me to the module tests which I'll add tonight or tomorrow before this is merged.

[ Full content available at: https://github.com/apache/couchdb/pull/1574 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [couchdb] nickva commented on pull request #1574: Fix builtin _sum reduce function

["nickva (GitHub)" <gi...@apache.org>]

Use `Overflowed` variable here as well, it also the updated correct logic for overflow.

To double check:

https://github.com/apache/couchdb/blob/bf72d619d11d787c4cd5bf343d7dfeb4b16ef01a/share/server/views.js#L41

[ Full content available at: https://github.com/apache/couchdb/pull/1574 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org