You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Stefan Roth (JIRA)" <ji...@apache.org> on 2011/05/17 16:34:47 UTC
[jira] [Created] (COUCHDB-1167) Importing several document and
adding attachments subsequently fails, and leads to crash report on server
(see unit test)
Importing several document and adding attachments subsequently fails, and leads to crash report on server (see unit test)
--------------------------------------------------------------------------------------------------------------------------
Key: COUCHDB-1167
URL: https://issues.apache.org/jira/browse/COUCHDB-1167
Project: CouchDB
Issue Type: Bug
Components: Database Core
Affects Versions: 1.0.2
Environment: Windows 7 and Mac OS X, Python CouchDB API
Reporter: Stefan Roth
Priority: Critical
I just started to explore CouchDB and implemented a few Python unit tests.
Perhaps I understand the concepts yet, but I observed a strange behavior when I try to add a series of documents, and subsequently want to add attachments to them. The CouchDB server produces a crash report (1), and some of the document attachments on the DB are empty afterwards (see test_insertDocsThenAttachments unittest (3) and output (2)).
(If I add the attachment immediately after the document is saved to the DB everything seems to be ok - see test_insertDocsAndAttachmentsInASingleLoop unittest).
Stefan
--------------------------------------------
(1)
=== CouchDB crash report ===
Starting CouchDB...
Eshell V5.8.1 (abort with ^G)
1> Apache CouchDB 1.0.2 (LogLevel=info) is starting.
1> Apache CouchDB has started. Time to relax.
1> [info] [<0.35.0>] Apache CouchDB has started on http://127.0.0.1:5984/
1> [info] [<0.107.0>] 127.0.0.1 - - 'DELETE' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211/doc1?rev=1-2a6731bb8fe08d8d55839abdafd3bd3c 201
1> [info] [<0.118.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1/doc2?rev=1-985759e4100dce92706deff32bdacefb 201
1> [info] [<0.129.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261/doc3?rev=1-ca7618164e9b385bc12b9de118f34450 201
1> [info] [<0.134.0>] checkpointing view update at seq 4 for testdb _temp
1> [info] [<0.134.0>] checkpointing view update at seq 6 for testdb _temp
1> [info] [<0.131.0>] 127.0.0.1 - - 'POST' /testdb/_temp_view 200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261 200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261/doc3 200
1> [error] [<0.131.0>] {error_report,<0.34.0>,
{<0.131.0>,crash_report,
[[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
{pid,<0.131.0>},
{registered_name,[]},
{error_info,
{error,
{badmatch,ok},
[{couch_httpd,handle_request_int,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_httpd,couch_secondary_services,couch_server_sup,<0.35.0>]},
{messages,[]},
{links,[<0.106.0>,#Port<0.1674>]},
{dictionary,
[{mochiweb_request_qs,[]},
{jsonp,undefined},
{mochiweb_request_cookie,[]}]},
{trap_exit,false},
{status,running},
{heap_size,987},
{stack_size,24},
{reductions,13653}],
[]]}}
1>
=CRASH REPORT==== 17-May-2011::16:17:16 ===
crasher:
initial call: mochiweb_socket_server:acceptor_loop/1
pid: <0.131.0>
registered_name: []
exception error: no match of right hand side value ok
in function couch_httpd:handle_request_int/5
in call from mochiweb_http:headers/5
ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
<0.35.0>]
messages: []
links: [<0.106.0>,#Port<0.1674>]
dictionary: [{mochiweb_request_qs,[]},
{jsonp,undefined},
{mochiweb_request_cookie,[]}]
trap_exit: false
status: running
heap_size: 987
stack_size: 24
reductions: 13653
neighbours:
1> [error] [<0.106.0>] {error_report,<0.34.0>,
{<0.106.0>,std_error,
{mochiweb_socket_server,235,{child_error,{badmatch,ok}}}}}
1>
=ERROR REPORT==== 17-May-2011::16:17:16 ===
{mochiweb_socket_server,235,{child_error,{badmatch,ok}}}
1>
(2)
=== Python unittest output (of failed test) ===
Finding files... done.
Importing test modules ... done.
<TestDocument u'e0abe6e707f34619b292eb152d812211'@u'1-2a6731bb8fe08d8d55839abdafd3bd3c' {'path': '.', 'name': 'doc1'}>
<TestDocument u'fb5b47d4e88f4943a94a46ed6bbad0a1'@u'1-985759e4100dce92706deff32bdacefb' {'path': '.', 'name': 'doc2'}>
<TestDocument u'b3880c2930704932872ee21309690261'@u'1-ca7618164e9b385bc12b9de118f34450' {'path': '.', 'name': 'doc3'}>
Unexpected error: <type 'exceptions.AttributeError'>
======================================================================
ERROR: test_insertDocsThenAttachments (couchdbbugreport.TestCouchDB)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 129, in test_insertDocsThenAttachments
self.exportdbdocs(self.db, tmpexportfolder)
File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 58, in exportdbdocs
self.exportdbdocattachment(db, row.key, root)
File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 70, in exportdbdocattachment
f.write(attachment.read())
AttributeError: 'NoneType' object has no attribute 'read'
----------------------------------------------------------------------
Ran 1 test in 0.673s
FAILED (errors=1)
(3)
=== Python unit test code ===
'''
Created on May 17, 2011
@author: sro
'''
import unittest
import couchdb
import filecmp
import shutil
import tempfile
import sys
import os
from couchdb import Document
from uuid import uuid4
from pprint import pprint
class TestDocument(Document):
def __init__(self):
self['_id'] = uuid4().hex
def getpath(self):
return self['path']
def setpath(self, value):
self['path'] = value
def getname(self):
return self['name']
def setname(self, value):
self['name'] = value
class TestCouchDB(unittest.TestCase):
def setUp(self):
# Change according to environment
serverurl = "http://admin:admin@localhost:5984"
dbname = "testdb"
self.db = self.initdb(serverurl, dbname)
def initdb(self, serverurl, dbname):
couch = couchdb.Server(serverurl)
# Create empty database for tests
try:
couch.delete(dbname);
except couchdb.http.ResourceNotFound:
print("DB did not exist")
finally:
couch.create(dbname);
db = couch[dbname]
return db
def exportdbdocs(self, db, root):
map_fun = '''function(doc) {
emit(doc._id, null);
}'''
for row in db.query(map_fun):
self.exportdbdocattachment(db, row.key, root)
def exportdbdocattachment(self, db, docid, root):
doc = db.get(docid)
attachment = db.get_attachment(doc.id, doc['name'])
targetfolder = os.path.join(root, doc['path'])
if not os.path.exists(targetfolder):
os.makedirs(targetfolder)
exportpath = os.path.join(targetfolder, doc['name'])
f = open(exportpath, 'w')
f.write(attachment.read())
attachment.close()
f.close
return exportpath
def createtestfile(self, path, size):
f = open(path, 'w')
f.write('\xff' * size)
f.close()
return f
def createfilecollection(self, tmpfolder):
path = []
path.append(os.path.join(tmpfolder, 'doc1'))
self.createtestfile(path[0], 100)
path.append(os.path.join(tmpfolder, 'doc2'))
self.createtestfile(path[1], 100)
path.append(os.path.join(tmpfolder, 'doc3'))
self.createtestfile(path[2], 100)
return path
def test_insertDocsAndAttachmentsInASingleLoop(self):
# This version of the test works
try:
docs = []
tmpfolder = tempfile.mkdtemp()
tmpexportfolder = tmpfolder + '-export'
# Here we first create all docs on the DB and then add attachments to them
paths = self.createfilecollection(tmpfolder)
for i in range(0, len(paths)):
# We append information about the initial attachment location
relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
filename = os.path.basename(paths[i])
doc = TestDocument()
doc.setpath(relpath)
doc.setname(filename)
docs.append(doc)
self.db.save(doc)
fp = open(paths[i])
pprint(doc)
self.db.put_attachment(doc, fp)
fp.close()
# We export the attachments again and compate them to the originals
self.exportdbdocs(self.db, tmpexportfolder)
self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))
except:
print "Unexpected error:", sys.exc_info()[0]
raise
finally:
shutil.rmtree(tmpfolder)
shutil.rmtree(tmpexportfolder)
def test_insertDocsThenAttachments(self):
# This version of the test crashes CouchDB
try:
docs = []
tmpfolder = tempfile.mkdtemp()
tmpexportfolder = tmpfolder + '-export'
# Here we first create all docs on the DB and then add attachments to them
paths = self.createfilecollection(tmpfolder)
for i in range(0, len(paths)):
# We append information about the initial attachment location
relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
filename = os.path.basename(paths[i])
doc = TestDocument()
doc.setpath(relpath)
doc.setname(filename)
docs.append(doc)
self.db.save(doc)
# Bulk insert (instead of self.db.save(docs[i])) crashes DB server as well
# self.db.update(docs)
for i in range(0, len(paths)):
doc = docs[i]
fp = open(paths[i])
pprint(doc)
self.db.put_attachment(doc, fp)
fp.close()
# We export the attachments and compare them to the originals
self.exportdbdocs(self.db, tmpexportfolder)
self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))
except:
print "Unexpected error:", sys.exc_info()[0]
raise
finally:
shutil.rmtree(tmpfolder)
shutil.rmtree(tmpexportfolder)
if __name__ == '__main__':
suite = unittest.TestSuite()
suite.addTest(TestCouchDB("test_insertDocsAndAttachmentsInASingleLoop"))
suite.addTest(TestCouchDB("test_insertDocsThenAttachments"))
unittest.TextTestRunner().run(suite)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1167) Importing several documents and
subsequently adding attachments fails and leads to crash report on DB
server (see unit test)
Posted by "Stefan Roth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefan Roth updated COUCHDB-1167:
---------------------------------
Description:
I just started to explore CouchDB and implemented a few Python unit tests.
Perhaps I don't understand the concepts yet, but I observed a strange behavior when I try to add a series of documents, and subsequently want to add attachments to them. The CouchDB server produces a crash report (1), and some of the document attachments on the DB are empty afterwards (see test_insertDocsThenAttachments unittest (3) and output (2)).
(If I add the attachment immediately after the document is saved to the DB everything seems to be ok - see test_insertDocsAndAttachmentsInASingleLoop unittest).
Stefan
--------------------------------------------
(1)
=== CouchDB crash report ===
Starting CouchDB...
Eshell V5.8.1 (abort with ^G)
1> Apache CouchDB 1.0.2 (LogLevel=info) is starting.
1> Apache CouchDB has started. Time to relax.
1> [info] [<0.35.0>] Apache CouchDB has started on http://127.0.0.1:5984/
1> [info] [<0.107.0>] 127.0.0.1 - - 'DELETE' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211/doc1?rev=1-2a6731bb8fe08d8d55839abdafd3bd3c 201
1> [info] [<0.118.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1/doc2?rev=1-985759e4100dce92706deff32bdacefb 201
1> [info] [<0.129.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261/doc3?rev=1-ca7618164e9b385bc12b9de118f34450 201
1> [info] [<0.134.0>] checkpointing view update at seq 4 for testdb _temp
1> [info] [<0.134.0>] checkpointing view update at seq 6 for testdb _temp
1> [info] [<0.131.0>] 127.0.0.1 - - 'POST' /testdb/_temp_view 200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261 200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261/doc3 200
1> [error] [<0.131.0>] {error_report,<0.34.0>,
{<0.131.0>,crash_report,
[[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
{pid,<0.131.0>},
{registered_name,[]},
{error_info,
{error,
{badmatch,ok},
[{couch_httpd,handle_request_int,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_httpd,couch_secondary_services,couch_server_sup,<0.35.0>]},
{messages,[]},
{links,[<0.106.0>,#Port<0.1674>]},
{dictionary,
[{mochiweb_request_qs,[]},
{jsonp,undefined},
{mochiweb_request_cookie,[]}]},
{trap_exit,false},
{status,running},
{heap_size,987},
{stack_size,24},
{reductions,13653}],
[]]}}
1>
=CRASH REPORT==== 17-May-2011::16:17:16 ===
crasher:
initial call: mochiweb_socket_server:acceptor_loop/1
pid: <0.131.0>
registered_name: []
exception error: no match of right hand side value ok
in function couch_httpd:handle_request_int/5
in call from mochiweb_http:headers/5
ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
<0.35.0>]
messages: []
links: [<0.106.0>,#Port<0.1674>]
dictionary: [{mochiweb_request_qs,[]},
{jsonp,undefined},
{mochiweb_request_cookie,[]}]
trap_exit: false
status: running
heap_size: 987
stack_size: 24
reductions: 13653
neighbours:
1> [error] [<0.106.0>] {error_report,<0.34.0>,
{<0.106.0>,std_error,
{mochiweb_socket_server,235,{child_error,{badmatch,ok}}}}}
1>
=ERROR REPORT==== 17-May-2011::16:17:16 ===
{mochiweb_socket_server,235,{child_error,{badmatch,ok}}}
1>
(2)
=== Python unittest output (of failed test) ===
Finding files... done.
Importing test modules ... done.
<TestDocument u'e0abe6e707f34619b292eb152d812211'@u'1-2a6731bb8fe08d8d55839abdafd3bd3c' {'path': '.', 'name': 'doc1'}>
<TestDocument u'fb5b47d4e88f4943a94a46ed6bbad0a1'@u'1-985759e4100dce92706deff32bdacefb' {'path': '.', 'name': 'doc2'}>
<TestDocument u'b3880c2930704932872ee21309690261'@u'1-ca7618164e9b385bc12b9de118f34450' {'path': '.', 'name': 'doc3'}>
Unexpected error: <type 'exceptions.AttributeError'>
======================================================================
ERROR: test_insertDocsThenAttachments (couchdbbugreport.TestCouchDB)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 129, in test_insertDocsThenAttachments
self.exportdbdocs(self.db, tmpexportfolder)
File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 58, in exportdbdocs
self.exportdbdocattachment(db, row.key, root)
File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 70, in exportdbdocattachment
f.write(attachment.read())
AttributeError: 'NoneType' object has no attribute 'read'
----------------------------------------------------------------------
Ran 1 test in 0.673s
FAILED (errors=1)
(3)
=== Python unit test code ===
'''
Created on May 17, 2011
@author: sro
'''
import unittest
import couchdb
import filecmp
import shutil
import tempfile
import sys
import os
from couchdb import Document
from uuid import uuid4
from pprint import pprint
class TestDocument(Document):
def __init__(self):
self['_id'] = uuid4().hex
def getpath(self):
return self['path']
def setpath(self, value):
self['path'] = value
def getname(self):
return self['name']
def setname(self, value):
self['name'] = value
class TestCouchDB(unittest.TestCase):
def setUp(self):
# Change according to environment
serverurl = "http://admin:admin@localhost:5984"
dbname = "testdb"
self.db = self.initdb(serverurl, dbname)
def initdb(self, serverurl, dbname):
couch = couchdb.Server(serverurl)
# Create empty database for tests
try:
couch.delete(dbname);
except couchdb.http.ResourceNotFound:
print("DB did not exist")
finally:
couch.create(dbname);
db = couch[dbname]
return db
def exportdbdocs(self, db, root):
map_fun = '''function(doc) {
emit(doc._id, null);
}'''
for row in db.query(map_fun):
self.exportdbdocattachment(db, row.key, root)
def exportdbdocattachment(self, db, docid, root):
doc = db.get(docid)
attachment = db.get_attachment(doc.id, doc['name'])
targetfolder = os.path.join(root, doc['path'])
if not os.path.exists(targetfolder):
os.makedirs(targetfolder)
exportpath = os.path.join(targetfolder, doc['name'])
f = open(exportpath, 'w')
f.write(attachment.read())
attachment.close()
f.close
return exportpath
def createtestfile(self, path, size):
f = open(path, 'w')
f.write('\xff' * size)
f.close()
return f
def createfilecollection(self, tmpfolder):
path = []
path.append(os.path.join(tmpfolder, 'doc1'))
self.createtestfile(path[0], 100)
path.append(os.path.join(tmpfolder, 'doc2'))
self.createtestfile(path[1], 100)
path.append(os.path.join(tmpfolder, 'doc3'))
self.createtestfile(path[2], 100)
return path
def test_insertDocsAndAttachmentsInASingleLoop(self):
# This version of the test works
try:
docs = []
tmpfolder = tempfile.mkdtemp()
tmpexportfolder = tmpfolder + '-export'
# Here we first create all docs on the DB and then add attachments to them
paths = self.createfilecollection(tmpfolder)
for i in range(0, len(paths)):
# We append information about the initial attachment location
relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
filename = os.path.basename(paths[i])
doc = TestDocument()
doc.setpath(relpath)
doc.setname(filename)
docs.append(doc)
self.db.save(doc)
fp = open(paths[i])
pprint(doc)
self.db.put_attachment(doc, fp)
fp.close()
# We export the attachments again and compate them to the originals
self.exportdbdocs(self.db, tmpexportfolder)
self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))
except:
print "Unexpected error:", sys.exc_info()[0]
raise
finally:
shutil.rmtree(tmpfolder)
shutil.rmtree(tmpexportfolder)
def test_insertDocsThenAttachments(self):
# This version of the test crashes CouchDB
try:
docs = []
tmpfolder = tempfile.mkdtemp()
tmpexportfolder = tmpfolder + '-export'
# Here we first create all docs on the DB and then add attachments to them
paths = self.createfilecollection(tmpfolder)
for i in range(0, len(paths)):
# We append information about the initial attachment location
relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
filename = os.path.basename(paths[i])
doc = TestDocument()
doc.setpath(relpath)
doc.setname(filename)
docs.append(doc)
self.db.save(doc)
# Bulk insert (instead of self.db.save(docs[i])) crashes DB server as well
# self.db.update(docs)
for i in range(0, len(paths)):
doc = docs[i]
fp = open(paths[i])
pprint(doc)
self.db.put_attachment(doc, fp)
fp.close()
# We export the attachments and compare them to the originals
self.exportdbdocs(self.db, tmpexportfolder)
self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))
except:
print "Unexpected error:", sys.exc_info()[0]
raise
finally:
shutil.rmtree(tmpfolder)
shutil.rmtree(tmpexportfolder)
if __name__ == '__main__':
suite = unittest.TestSuite()
suite.addTest(TestCouchDB("test_insertDocsAndAttachmentsInASingleLoop"))
suite.addTest(TestCouchDB("test_insertDocsThenAttachments"))
unittest.TextTestRunner().run(suite)
was:
I just started to explore CouchDB and implemented a few Python unit tests.
Perhaps I understand the concepts yet, but I observed a strange behavior when I try to add a series of documents, and subsequently want to add attachments to them. The CouchDB server produces a crash report (1), and some of the document attachments on the DB are empty afterwards (see test_insertDocsThenAttachments unittest (3) and output (2)).
(If I add the attachment immediately after the document is saved to the DB everything seems to be ok - see test_insertDocsAndAttachmentsInASingleLoop unittest).
Stefan
--------------------------------------------
(1)
=== CouchDB crash report ===
Starting CouchDB...
Eshell V5.8.1 (abort with ^G)
1> Apache CouchDB 1.0.2 (LogLevel=info) is starting.
1> Apache CouchDB has started. Time to relax.
1> [info] [<0.35.0>] Apache CouchDB has started on http://127.0.0.1:5984/
1> [info] [<0.107.0>] 127.0.0.1 - - 'DELETE' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211/doc1?rev=1-2a6731bb8fe08d8d55839abdafd3bd3c 201
1> [info] [<0.118.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1/doc2?rev=1-985759e4100dce92706deff32bdacefb 201
1> [info] [<0.129.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261/doc3?rev=1-ca7618164e9b385bc12b9de118f34450 201
1> [info] [<0.134.0>] checkpointing view update at seq 4 for testdb _temp
1> [info] [<0.134.0>] checkpointing view update at seq 6 for testdb _temp
1> [info] [<0.131.0>] 127.0.0.1 - - 'POST' /testdb/_temp_view 200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261 200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261/doc3 200
1> [error] [<0.131.0>] {error_report,<0.34.0>,
{<0.131.0>,crash_report,
[[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
{pid,<0.131.0>},
{registered_name,[]},
{error_info,
{error,
{badmatch,ok},
[{couch_httpd,handle_request_int,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_httpd,couch_secondary_services,couch_server_sup,<0.35.0>]},
{messages,[]},
{links,[<0.106.0>,#Port<0.1674>]},
{dictionary,
[{mochiweb_request_qs,[]},
{jsonp,undefined},
{mochiweb_request_cookie,[]}]},
{trap_exit,false},
{status,running},
{heap_size,987},
{stack_size,24},
{reductions,13653}],
[]]}}
1>
=CRASH REPORT==== 17-May-2011::16:17:16 ===
crasher:
initial call: mochiweb_socket_server:acceptor_loop/1
pid: <0.131.0>
registered_name: []
exception error: no match of right hand side value ok
in function couch_httpd:handle_request_int/5
in call from mochiweb_http:headers/5
ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
<0.35.0>]
messages: []
links: [<0.106.0>,#Port<0.1674>]
dictionary: [{mochiweb_request_qs,[]},
{jsonp,undefined},
{mochiweb_request_cookie,[]}]
trap_exit: false
status: running
heap_size: 987
stack_size: 24
reductions: 13653
neighbours:
1> [error] [<0.106.0>] {error_report,<0.34.0>,
{<0.106.0>,std_error,
{mochiweb_socket_server,235,{child_error,{badmatch,ok}}}}}
1>
=ERROR REPORT==== 17-May-2011::16:17:16 ===
{mochiweb_socket_server,235,{child_error,{badmatch,ok}}}
1>
(2)
=== Python unittest output (of failed test) ===
Finding files... done.
Importing test modules ... done.
<TestDocument u'e0abe6e707f34619b292eb152d812211'@u'1-2a6731bb8fe08d8d55839abdafd3bd3c' {'path': '.', 'name': 'doc1'}>
<TestDocument u'fb5b47d4e88f4943a94a46ed6bbad0a1'@u'1-985759e4100dce92706deff32bdacefb' {'path': '.', 'name': 'doc2'}>
<TestDocument u'b3880c2930704932872ee21309690261'@u'1-ca7618164e9b385bc12b9de118f34450' {'path': '.', 'name': 'doc3'}>
Unexpected error: <type 'exceptions.AttributeError'>
======================================================================
ERROR: test_insertDocsThenAttachments (couchdbbugreport.TestCouchDB)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 129, in test_insertDocsThenAttachments
self.exportdbdocs(self.db, tmpexportfolder)
File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 58, in exportdbdocs
self.exportdbdocattachment(db, row.key, root)
File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 70, in exportdbdocattachment
f.write(attachment.read())
AttributeError: 'NoneType' object has no attribute 'read'
----------------------------------------------------------------------
Ran 1 test in 0.673s
FAILED (errors=1)
(3)
=== Python unit test code ===
'''
Created on May 17, 2011
@author: sro
'''
import unittest
import couchdb
import filecmp
import shutil
import tempfile
import sys
import os
from couchdb import Document
from uuid import uuid4
from pprint import pprint
class TestDocument(Document):
def __init__(self):
self['_id'] = uuid4().hex
def getpath(self):
return self['path']
def setpath(self, value):
self['path'] = value
def getname(self):
return self['name']
def setname(self, value):
self['name'] = value
class TestCouchDB(unittest.TestCase):
def setUp(self):
# Change according to environment
serverurl = "http://admin:admin@localhost:5984"
dbname = "testdb"
self.db = self.initdb(serverurl, dbname)
def initdb(self, serverurl, dbname):
couch = couchdb.Server(serverurl)
# Create empty database for tests
try:
couch.delete(dbname);
except couchdb.http.ResourceNotFound:
print("DB did not exist")
finally:
couch.create(dbname);
db = couch[dbname]
return db
def exportdbdocs(self, db, root):
map_fun = '''function(doc) {
emit(doc._id, null);
}'''
for row in db.query(map_fun):
self.exportdbdocattachment(db, row.key, root)
def exportdbdocattachment(self, db, docid, root):
doc = db.get(docid)
attachment = db.get_attachment(doc.id, doc['name'])
targetfolder = os.path.join(root, doc['path'])
if not os.path.exists(targetfolder):
os.makedirs(targetfolder)
exportpath = os.path.join(targetfolder, doc['name'])
f = open(exportpath, 'w')
f.write(attachment.read())
attachment.close()
f.close
return exportpath
def createtestfile(self, path, size):
f = open(path, 'w')
f.write('\xff' * size)
f.close()
return f
def createfilecollection(self, tmpfolder):
path = []
path.append(os.path.join(tmpfolder, 'doc1'))
self.createtestfile(path[0], 100)
path.append(os.path.join(tmpfolder, 'doc2'))
self.createtestfile(path[1], 100)
path.append(os.path.join(tmpfolder, 'doc3'))
self.createtestfile(path[2], 100)
return path
def test_insertDocsAndAttachmentsInASingleLoop(self):
# This version of the test works
try:
docs = []
tmpfolder = tempfile.mkdtemp()
tmpexportfolder = tmpfolder + '-export'
# Here we first create all docs on the DB and then add attachments to them
paths = self.createfilecollection(tmpfolder)
for i in range(0, len(paths)):
# We append information about the initial attachment location
relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
filename = os.path.basename(paths[i])
doc = TestDocument()
doc.setpath(relpath)
doc.setname(filename)
docs.append(doc)
self.db.save(doc)
fp = open(paths[i])
pprint(doc)
self.db.put_attachment(doc, fp)
fp.close()
# We export the attachments again and compate them to the originals
self.exportdbdocs(self.db, tmpexportfolder)
self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))
except:
print "Unexpected error:", sys.exc_info()[0]
raise
finally:
shutil.rmtree(tmpfolder)
shutil.rmtree(tmpexportfolder)
def test_insertDocsThenAttachments(self):
# This version of the test crashes CouchDB
try:
docs = []
tmpfolder = tempfile.mkdtemp()
tmpexportfolder = tmpfolder + '-export'
# Here we first create all docs on the DB and then add attachments to them
paths = self.createfilecollection(tmpfolder)
for i in range(0, len(paths)):
# We append information about the initial attachment location
relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
filename = os.path.basename(paths[i])
doc = TestDocument()
doc.setpath(relpath)
doc.setname(filename)
docs.append(doc)
self.db.save(doc)
# Bulk insert (instead of self.db.save(docs[i])) crashes DB server as well
# self.db.update(docs)
for i in range(0, len(paths)):
doc = docs[i]
fp = open(paths[i])
pprint(doc)
self.db.put_attachment(doc, fp)
fp.close()
# We export the attachments and compare them to the originals
self.exportdbdocs(self.db, tmpexportfolder)
self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))
except:
print "Unexpected error:", sys.exc_info()[0]
raise
finally:
shutil.rmtree(tmpfolder)
shutil.rmtree(tmpexportfolder)
if __name__ == '__main__':
suite = unittest.TestSuite()
suite.addTest(TestCouchDB("test_insertDocsAndAttachmentsInASingleLoop"))
suite.addTest(TestCouchDB("test_insertDocsThenAttachments"))
unittest.TextTestRunner().run(suite)
Summary: Importing several documents and subsequently adding attachments fails and leads to crash report on DB server (see unit test) (was: Importing several document and adding attachments subsequently fails, and leads to crash report on server (see unit test) )
> Importing several documents and subsequently adding attachments fails and leads to crash report on DB server (see unit test)
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: COUCHDB-1167
> URL: https://issues.apache.org/jira/browse/COUCHDB-1167
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
> Affects Versions: 1.0.2
> Environment: Windows 7 and Mac OS X, Python CouchDB API
> Reporter: Stefan Roth
> Priority: Critical
> Labels: attachments, crash
>
> I just started to explore CouchDB and implemented a few Python unit tests.
> Perhaps I don't understand the concepts yet, but I observed a strange behavior when I try to add a series of documents, and subsequently want to add attachments to them. The CouchDB server produces a crash report (1), and some of the document attachments on the DB are empty afterwards (see test_insertDocsThenAttachments unittest (3) and output (2)).
> (If I add the attachment immediately after the document is saved to the DB everything seems to be ok - see test_insertDocsAndAttachmentsInASingleLoop unittest).
> Stefan
> --------------------------------------------
> (1)
> === CouchDB crash report ===
> Starting CouchDB...
> Eshell V5.8.1 (abort with ^G)
> 1> Apache CouchDB 1.0.2 (LogLevel=info) is starting.
> 1> Apache CouchDB has started. Time to relax.
> 1> [info] [<0.35.0>] Apache CouchDB has started on http://127.0.0.1:5984/
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'DELETE' /testdb 200
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb 201
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211 201
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1 201
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261 201
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211/doc1?rev=1-2a6731bb8fe08d8d55839abdafd3bd3c 201
> 1> [info] [<0.118.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1/doc2?rev=1-985759e4100dce92706deff32bdacefb 201
> 1> [info] [<0.129.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261/doc3?rev=1-ca7618164e9b385bc12b9de118f34450 201
> 1> [info] [<0.134.0>] checkpointing view update at seq 4 for testdb _temp
> 1> [info] [<0.134.0>] checkpointing view update at seq 6 for testdb _temp
> 1> [info] [<0.131.0>] 127.0.0.1 - - 'POST' /testdb/_temp_view 200
> 1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261 200
> 1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261/doc3 200
> 1> [error] [<0.131.0>] {error_report,<0.34.0>,
> {<0.131.0>,crash_report,
> [[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
> {pid,<0.131.0>},
> {registered_name,[]},
> {error_info,
> {error,
> {badmatch,ok},
> [{couch_httpd,handle_request_int,5},
> {mochiweb_http,headers,5},
> {proc_lib,init_p_do_apply,3}]}},
> {ancestors,
> [couch_httpd,couch_secondary_services,couch_server_sup,<0.35.0>]},
> {messages,[]},
> {links,[<0.106.0>,#Port<0.1674>]},
> {dictionary,
> [{mochiweb_request_qs,[]},
> {jsonp,undefined},
> {mochiweb_request_cookie,[]}]},
> {trap_exit,false},
> {status,running},
> {heap_size,987},
> {stack_size,24},
> {reductions,13653}],
> []]}}
> 1>
> =CRASH REPORT==== 17-May-2011::16:17:16 ===
> crasher:
> initial call: mochiweb_socket_server:acceptor_loop/1
> pid: <0.131.0>
> registered_name: []
> exception error: no match of right hand side value ok
> in function couch_httpd:handle_request_int/5
> in call from mochiweb_http:headers/5
> ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
> <0.35.0>]
> messages: []
> links: [<0.106.0>,#Port<0.1674>]
> dictionary: [{mochiweb_request_qs,[]},
> {jsonp,undefined},
> {mochiweb_request_cookie,[]}]
> trap_exit: false
> status: running
> heap_size: 987
> stack_size: 24
> reductions: 13653
> neighbours:
> 1> [error] [<0.106.0>] {error_report,<0.34.0>,
> {<0.106.0>,std_error,
> {mochiweb_socket_server,235,{child_error,{badmatch,ok}}}}}
> 1>
> =ERROR REPORT==== 17-May-2011::16:17:16 ===
> {mochiweb_socket_server,235,{child_error,{badmatch,ok}}}
> 1>
> (2)
> === Python unittest output (of failed test) ===
> Finding files... done.
> Importing test modules ... done.
> <TestDocument u'e0abe6e707f34619b292eb152d812211'@u'1-2a6731bb8fe08d8d55839abdafd3bd3c' {'path': '.', 'name': 'doc1'}>
> <TestDocument u'fb5b47d4e88f4943a94a46ed6bbad0a1'@u'1-985759e4100dce92706deff32bdacefb' {'path': '.', 'name': 'doc2'}>
> <TestDocument u'b3880c2930704932872ee21309690261'@u'1-ca7618164e9b385bc12b9de118f34450' {'path': '.', 'name': 'doc3'}>
> Unexpected error: <type 'exceptions.AttributeError'>
> ======================================================================
> ERROR: test_insertDocsThenAttachments (couchdbbugreport.TestCouchDB)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 129, in test_insertDocsThenAttachments
> self.exportdbdocs(self.db, tmpexportfolder)
> File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 58, in exportdbdocs
> self.exportdbdocattachment(db, row.key, root)
> File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 70, in exportdbdocattachment
> f.write(attachment.read())
> AttributeError: 'NoneType' object has no attribute 'read'
> ----------------------------------------------------------------------
> Ran 1 test in 0.673s
> FAILED (errors=1)
> (3)
> === Python unit test code ===
> '''
> Created on May 17, 2011
> @author: sro
> '''
> import unittest
> import couchdb
> import filecmp
> import shutil
> import tempfile
> import sys
> import os
> from couchdb import Document
> from uuid import uuid4
> from pprint import pprint
> class TestDocument(Document):
> def __init__(self):
> self['_id'] = uuid4().hex
>
> def getpath(self):
> return self['path']
> def setpath(self, value):
> self['path'] = value
>
> def getname(self):
> return self['name']
>
> def setname(self, value):
> self['name'] = value
> class TestCouchDB(unittest.TestCase):
> def setUp(self):
> # Change according to environment
> serverurl = "http://admin:admin@localhost:5984"
> dbname = "testdb"
> self.db = self.initdb(serverurl, dbname)
>
> def initdb(self, serverurl, dbname):
> couch = couchdb.Server(serverurl)
> # Create empty database for tests
> try:
> couch.delete(dbname);
> except couchdb.http.ResourceNotFound:
> print("DB did not exist")
> finally:
> couch.create(dbname);
> db = couch[dbname]
> return db
> def exportdbdocs(self, db, root):
> map_fun = '''function(doc) {
> emit(doc._id, null);
> }'''
> for row in db.query(map_fun):
> self.exportdbdocattachment(db, row.key, root)
> def exportdbdocattachment(self, db, docid, root):
> doc = db.get(docid)
> attachment = db.get_attachment(doc.id, doc['name'])
>
> targetfolder = os.path.join(root, doc['path'])
> if not os.path.exists(targetfolder):
> os.makedirs(targetfolder)
> exportpath = os.path.join(targetfolder, doc['name'])
>
> f = open(exportpath, 'w')
> f.write(attachment.read())
> attachment.close()
> f.close
>
> return exportpath
> def createtestfile(self, path, size):
> f = open(path, 'w')
> f.write('\xff' * size)
> f.close()
> return f
>
> def createfilecollection(self, tmpfolder):
> path = []
>
> path.append(os.path.join(tmpfolder, 'doc1'))
> self.createtestfile(path[0], 100)
>
> path.append(os.path.join(tmpfolder, 'doc2'))
> self.createtestfile(path[1], 100)
>
> path.append(os.path.join(tmpfolder, 'doc3'))
> self.createtestfile(path[2], 100)
>
> return path
>
> def test_insertDocsAndAttachmentsInASingleLoop(self):
> # This version of the test works
> try:
> docs = []
> tmpfolder = tempfile.mkdtemp()
> tmpexportfolder = tmpfolder + '-export'
>
> # Here we first create all docs on the DB and then add attachments to them
> paths = self.createfilecollection(tmpfolder)
> for i in range(0, len(paths)):
> # We append information about the initial attachment location
> relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
> filename = os.path.basename(paths[i])
>
> doc = TestDocument()
> doc.setpath(relpath)
> doc.setname(filename)
> docs.append(doc)
>
> self.db.save(doc)
> fp = open(paths[i])
> pprint(doc)
> self.db.put_attachment(doc, fp)
> fp.close()
>
> # We export the attachments again and compate them to the originals
> self.exportdbdocs(self.db, tmpexportfolder)
> self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))
> except:
> print "Unexpected error:", sys.exc_info()[0]
> raise
>
> finally:
> shutil.rmtree(tmpfolder)
> shutil.rmtree(tmpexportfolder)
> def test_insertDocsThenAttachments(self):
> # This version of the test crashes CouchDB
> try:
> docs = []
> tmpfolder = tempfile.mkdtemp()
> tmpexportfolder = tmpfolder + '-export'
>
> # Here we first create all docs on the DB and then add attachments to them
> paths = self.createfilecollection(tmpfolder)
> for i in range(0, len(paths)):
> # We append information about the initial attachment location
> relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
> filename = os.path.basename(paths[i])
>
> doc = TestDocument()
> doc.setpath(relpath)
> doc.setname(filename)
> docs.append(doc)
>
> self.db.save(doc)
> # Bulk insert (instead of self.db.save(docs[i])) crashes DB server as well
> # self.db.update(docs)
>
> for i in range(0, len(paths)):
> doc = docs[i]
> fp = open(paths[i])
> pprint(doc)
> self.db.put_attachment(doc, fp)
> fp.close()
>
> # We export the attachments and compare them to the originals
> self.exportdbdocs(self.db, tmpexportfolder)
> self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))
> except:
> print "Unexpected error:", sys.exc_info()[0]
> raise
>
> finally:
> shutil.rmtree(tmpfolder)
> shutil.rmtree(tmpexportfolder)
>
> if __name__ == '__main__':
> suite = unittest.TestSuite()
> suite.addTest(TestCouchDB("test_insertDocsAndAttachmentsInASingleLoop"))
> suite.addTest(TestCouchDB("test_insertDocsThenAttachments"))
> unittest.TextTestRunner().run(suite)
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira