You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Charles Woerner <CW...@demandbase.com> on 2010/03/11 00:09:42 UTC
c++ broker startup problem with large journal containing lots of data
Hello,
I've run into a problem running some tests to probe various aspects of
QPID when all consumers have been down for extended periods of time (2
hours). I'm enqueuing data at a steady rate of about 450 enqueues per
second for 2 hours. My queue is a large durable queue (~ 80GB) using
the msgstore persistence mechanism. I have set max-session-rate=0 and
limit-policy=flow-to-disk. I should point out that QPID has been
running smoothly for 6 months, so I'm confident that the system is
stable under it's normal load. Also, my expectation based on my past
experience with QPID is that something is simply misconfigured, not
that there is a bug in QPID. We're using QPID 0.5.752581-34 provided
by RedHat (MRG).
The two problems I'm seeing are:
1) After enqueuing 2 hours worth of data at 450 enqueues/sec I am
unable to startup the broker after cleanly shutting it down. After
~10 minutes into the recovery sequence at startup (after Journal
recovery phase 1) it fails with the message "Broker startup failed:
cannot read from child process". Below is the relevant trace output
from the log.
2) qpid seems to be using more memory than I expected. I have
configured the max-queue-size as 108576000 (100x the default of 10MB,
or 1GB) with a policy of flow-to-disk. My understanding is that flow-
to-disk should allow me to continue to accept and serialize enqueues
beyond the max-queue-size limit and that everything up to the ${max-
queue-size} amount of data will be held in memory. Indeed, it appears
to work that way, best I can tell. Given those settings and my
experience with QPID I expected to see a little over 1 GB of memory
being used, but I see it's way more - over 1.6 GB of memory. Is there
a simple formula for approximating how much memory a given QPID
instance will use based on the number of queues and various
configuration options, or should I just experiment?
]$ top
...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7699 qpidd 16 0 2548m 1.6g 1408 S 0.3 94.0 7:39.27 qpidd
...
]$ free
total used free shared buffers
cached
Mem: 1747764 1739964 7800 0 116
7580
-/+ buffers/cache: 1732268 15496
Swap: 917496 909268 8228
I would be very thankful if someone more knowledgeable than I about
QPID were able to take a look at my configuration and help me
determine what the problem is and how to work around it. Thanks in
advance.
Our qpidd.conf file is:
daemon=yes
log-to-file=/var/log/qpid/sfbroker.log
load-module=/usr/lib/qpid/daemon/msgstore.so
data-dir=/vol/qpidd
num-jfiles=64
jfile-size-pgs=20480
wait=1800
auth=no
trace=yes
max-session-rate=0
The broker queue is called "hits" and was created using the following
code. My understanding is that max-queue-size + flow-to-disk should
give me approximately 1GB of queue data in memory, with the older
items flowing to a disk store. The actual limit of the entire queue
should be limited to 80% of the journal size. If anyone is obliged
I'd love some feedback as to whether alternate disk geometries would
provide for a faster queue setup, recovery, and startup time.
qpid-config add queue hits --durable --file-count 64 --file-size
20480 --max-queue-size 1048576000 --limit-policy flow-to-disk
qpid-config bind amq.direct hits /hits
A trace of the log at startup follows:
Starting Qpid AMQP daemon: 2010-mar-10 22:02:12 info Loaded Module: /
usr/lib/qpid/daemon/msgstore.so
2010-mar-10 22:02:12 info Loaded Module: /usr/lib/qpid/daemon/
replicating_listener.so
2010-mar-10 22:02:12 info Loaded Module: /usr/lib/qpid/daemon/ssl.so
2010-mar-10 22:02:12 info Loaded Module: /usr/lib/qpid/daemon/
msgstore.so
2010-mar-10 22:02:12 info Loaded Module: /usr/lib/qpid/daemon/
replicating_listener.so
2010-mar-10 22:02:12 info Loaded Module: /usr/lib/qpid/daemon/ssl.so
2010-mar-10 22:02:13 info Loaded Module: /usr/lib/qpid/daemon/xml.so
2010-mar-10 22:02:13 info Loaded Module: /usr/lib/qpid/daemon/
replication_exchange.so
2010-mar-10 22:02:13 info Loaded Module: /usr/lib/qpid/daemon/
msgstore.so
2010-mar-10 22:02:13 info SSL connector not enabled, you must set
QPID_SSL_CERT_DB to enable it.
2010-mar-10 22:02:13 info Loaded Module: /usr/lib/qpid/client/
sslconnector.so
2010-mar-10 22:02:13 info Loaded Module: /usr/lib/qpid/daemon/cluster.so
2010-mar-10 22:02:13 info Loaded Module: /usr/lib/qpid/daemon/xml.so
2010-mar-10 22:02:13 info Loaded Module: /usr/lib/qpid/daemon/
replication_exchange.so
2010-mar-10 22:02:13 info Loaded Module: /usr/lib/qpid/daemon/
msgstore.so
2010-mar-10 22:02:13 info SSL connector not enabled, you must set
QPID_SSL_CERT_DB to enable it.
2010-mar-10 22:02:13 info Loaded Module: /usr/lib/qpid/client/
sslconnector.so
2010-mar-10 22:02:13 info Loaded Module: /usr/lib/qpid/daemon/cluster.so
2010-mar-10 22:02:13 debug Forked daemon child process
2010-mar-10 22:02:13 info Management enabled
2010-mar-10 22:02:13 debug ManagementBroker restored broker ID:
d87970a3-39b8-41ae-bf7f-38c460ae9795
2010-mar-10 22:02:13 debug ManagementBroker boot sequence: 6
2010-mar-10 22:02:13 debug ManagementBroker added package
org.apache.qpid.broker
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:system
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:broker
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:agent
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:vhost
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:queue
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:exchange
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:binding
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:connection
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:link
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:bridge
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:session
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:clientConnect
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:clientConnectFail
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:clientDisconnect
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:brokerLinkUp
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:brokerLinkDown
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:queueDeclare
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:queueDelete
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:exchangeDeclare
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:exchangeDelete
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:bind
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:unbind
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:subscribe
2010-mar-10 22:02:13 debug ManagementBroker added class
org.apache.qpid.broker:unsubscribe
2010-mar-10 22:02:13 notice Journal "TplStore": Created
2010-mar-10 22:02:13 debug Journal "TplStore": Journal directory = "/
vol/qpidd/rhm/tpl/"; Base file name = "tpl"
2010-mar-10 22:02:13 notice Store module initialized; dir=/vol/qpidd
2010-mar-10 22:02:13 info > Default files per journal: 64
2010-mar-10 22:02:13 info > Default journal file size: 20480 (wpgs)
2010-mar-10 22:02:13 info > Default write cache page size: 32 (Kib)
2010-mar-10 22:02:13 info > Default number of write cache pages: 32
2010-mar-10 22:02:13 info > TPL files per journal: 8
2010-mar-10 22:02:13 info > TPL journal file size: 24 (wpgs)
2010-mar-10 22:02:13 info > TPL write cache page size: 4 (Kib)
2010-mar-10 22:02:13 info > TPL number of write cache pages: 64
2010-mar-10 22:02:13 debug Configured queue hits with no-local=0
2010-mar-10 22:02:13 debug Configured queue hits with qpid.trace.id=''
and qpid.trace.exclude='' i.e. 0 elements
2010-mar-10 22:02:13 notice Journal "hits": Created
2010-mar-10 22:02:13 debug Journal "hits": Journal directory = "/vol/
qpidd/rhm/jrnl/000c/hits/"; Base file name = "JournalData"
2010-mar-10 22:02:13 debug Journal "hits": Recover; num_jfiles=64
jfsize_sblks=2621440 queue_id = 0x2 wcache_pgsize_sblks=64
wcache_num_pages=32
2010-mar-10 22:09:59 debug Journal "hits": Recover file analysis
(jid="hits"): njf=64 ae=F aemjf=0 owi=F frot=T jempty=F ffid=0
fro=0x200 (4 dblks) lfid=2 eo=0x3a1fc800 (7618448 dblks)
h_rid=0x4322eb lffull=F jfull=F Enqueued records (txn & non-txn):
[ fid_00=0 fid_01=1208884 fid_02=903881 fid_03=0 fid_04=0 fid_05=0
fid_06=0 fid_07=0 fid_08=0 fid_09=0 fid_10=0 fid_11=0 fid_12=0
fid_13=0 fid_14=0 fid_15=0 fid_16=0 fid_17=0 fid_18=0 fid_19=0
fid_20=0 fid_21=0 fid_22=0 fid_23=0 fid_24=0 fid_25=0 fid_26=0
fid_27=0 fid_28=0 fid_29=0 fid_30=0 fid_31=0 fid_32=0 fid_33=0
fid_34=0 fid_35=0 fid_36=0 fid_37=0 fid_38=0 fid_39=0 fid_40=0
fid_41=0 fid_42=0 fid_43=0 fid_44=0 fid_45=0 fid_46=0 fid_47=0
fid_48=0 fid_49=0 fid_50=0 fid_51=0 fid_52=0 fid_53=0 fid_54=0
fid_55=0 fid_56=0 fid_57=0 fid_58=0 fid_59=0 fid_60=0 fid_61=0
fid_62=0 fid_63=0 ]
2010-mar-10 22:09:59 debug Journal "hits": Recover phase 1 complete;
highest rid found = 0x4322eb; emap.size=2112765; tmap.size=0; journal
now read-only.
2010-mar-10 22:11:44 debug Exception constructed: Cannot read from
child process.
2010-mar-10 22:11:44 critical Broker start-up failed: Cannot read from
child process.
__
Charles Woerner | cwoerner@demandbase.com | demandbase |
415.683.2669
Re: c++ broker startup problem with large journal containing lots of data
Posted by Charles Woerner <cw...@demandbase.com>.
To follow up, it always fails at startup with this large queue, but I
don't always see the "Broker startup failed: cannot read from child
process" error. Sometimes it just dies.
__
Charles Woerner | cwoerner@demandbase.com | demandbase |
415.683.2669
Re: c++ broker startup problem with large journal containing lots of data
Posted by Charles Woerner <cw...@demandbase.com>.
Increasing the amount of memory available allowed me to complete
startup phase. Qpid is 3.2g resident in memory, which, again, is a
lot more than I expected.
Charles Woerner | cwoerner@demandbase.com | demandbase
Re: c++ broker startup problem with large journal containing lots of data
Posted by Charles Woerner <cw...@demandbase.com>.
Kim,
Thank you for your detailed follow up. Sounds like qpid basically
does what I want. We have no such inter-queue dependencies.
Actually, I think I read somewhere that this question about how to
handle policy when a message is delivered across multiple queues was
the result of a gap in the AMQP 10 spec.
Charles Woerner | cwoerner@demandbase.com | demandbase
Re: c++ broker startup problem with large journal containing lots
of data
Posted by Kim van der Riet <ki...@redhat.com>.
On Thu, 2010-03-11 at 13:13 -0800, Charles Woerner wrote:
> Thanks Kim,
>
> 1) I'm using qpid from RedHat MRG 0.5.752581-34 for i386 and the store
> module is RHM 0.5.3206-27, also for i386.
Thanks, this is the most recent set.
>
> 2) Yes, my messages are persistent. I am aware that transient
> messages are discarded on recovery. Your comment that durable
> messages "have their message content discarded when the policy limit
> is reached" is surprising. I have an 80 GB store and a 1 GB queue
> with flow-to-disk. My expectation was that I would be able to
> continue to enqueue durable messages past the 1GB mark (up to 80% of
> 80 GB) and only the first (or latest, not sure) 1GB of queue data
> would be held in memory. I expected the dequeues would sort of
> "backfill" the 1GB memory buffer until the "flowed-to-disk" messages
> were consumed from the store. I recognize that that this is sort of
> dual-purposing the backing store to do both journaling and spillover,
> but that doesn't seem too far fetched. I'm beginning to think this is
> incorrect. Is there some kind of identity relating available memory
> to store geometry to queue size (ie. is the following a good rule of
> thumb "available memory" = "store size" = "sum(max-queue-size of all
> queues)")?
Perhaps there is a terminology gap... By "discarded", I mean the process
by which the broker discards the message content from memory while
retaining the message metadata in the queue - this is the process known
as "flow-to-disk". In order to consume a message which has has its
content discarded, the store must be called upon to read the content
from disk to restore the content. So you are correct: when the policy
threshold is reached, messages may continue to be enqueued, but with
their message content saved to disk and discarded (from memory).
You are also correct concerning the store space limitations - being
essentially a circular disk store, the size of the journal files impose
a fundamental limit on the total size of the content that may be stored.
When an approx. 80% full condition is reached, the store will deny
further storage for enqueue operations, dequeues may continue as normal.
The dequeues (if they consume the messages in approximately the same
order as they were enqueued) will free up space in the journals,
allowing enqueuing to continue as normal. The store does not distinguish
between persistent and transient messages placed there (except in
recovery in which the transient messages are discarded); as messages are
consumed, the disk space is freed up in the journal.
There is a distinct performance hole that is entered when flow-to-disk
is triggered. This process requires that the store perform both read and
write operations simultaneously, which causes the disk head to jump
around repeatedly within the journal files. Normal store operation is
sequential write-only.
I should point out that the concept of "flow-to-disk" as implemented is
not well thought through and has several noticeable limitations:
a) All policies are per-queue only, there is not a means to cleanly set
an overall limit. There is no limit on creating additional queues when
memory is near fully used;
b) There are several complicated inter-queue interactions. If a message
is routed to several queues (bearing in mind that there is just one copy
of the message content in memory, and that the store is created on a
per-queue basis), and a size policy is violated on just one of those
queues, then it is not possible to discard the message content in this
case as the other queues will not have knowledge of the missing message
content in their stores.
This is one area of the implementation that needs a re-think. What we
currently have is somewhat useful if you can live with its limitations.
Gordon may care to comment further on this.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: c++ broker startup problem with large journal containing lots of data
Posted by Charles Woerner <cw...@demandbase.com>.
Thanks Kim,
1) I'm using qpid from RedHat MRG 0.5.752581-34 for i386 and the store
module is RHM 0.5.3206-27, also for i386.
2) Yes, my messages are persistent. I am aware that transient
messages are discarded on recovery. Your comment that durable
messages "have their message content discarded when the policy limit
is reached" is surprising. I have an 80 GB store and a 1 GB queue
with flow-to-disk. My expectation was that I would be able to
continue to enqueue durable messages past the 1GB mark (up to 80% of
80 GB) and only the first (or latest, not sure) 1GB of queue data
would be held in memory. I expected the dequeues would sort of
"backfill" the 1GB memory buffer until the "flowed-to-disk" messages
were consumed from the store. I recognize that that this is sort of
dual-purposing the backing store to do both journaling and spillover,
but that doesn't seem too far fetched. I'm beginning to think this is
incorrect. Is there some kind of identity relating available memory
to store geometry to queue size (ie. is the following a good rule of
thumb "available memory" = "store size" = "sum(max-queue-size of all
queues)")?
Also, in an attempt to (slowly) recover the queue I allocated 2 GB
swap. This staved off the "Cannot read from child process" error with
a c++ bad_alloc exception which I could see as a debug statement in
the log. However, 2 hours later the system was still in the same
unresponsive state (seemingly trying to complete recovery). It had
not yet begun listening on 5672. I captured the lsof for qpidd user
in case that might also be helpful. Yes, the sfbroker.log file is big
and no, truncating it doesn't alleviate the problem :)
~]# lsof -u qpidd
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
qpidd 12365 qpidd cwd DIR 8,1 4096 573441 /root
qpidd 12365 qpidd rtd DIR 8,1 4096 2 /
qpidd 12365 qpidd txt REG 8,1 76165 1299146 /usr/
sbin/qpidd
qpidd 12365 qpidd mem REG 8,1 18031759 1215641 /usr/
lib/libqpidcommon.so.0.1.0
qpidd 12365 qpidd mem REG 8,1 14644 786449 /lib/
libdl-2.5.so
qpidd 12365 qpidd mem REG 8,1 97512 1214114 /usr/
lib/libsasl2.so.2.0.22
qpidd 12365 qpidd mem REG 8,1 206396 786499 /lib/
i686/nosegneg/libm-2.5.so
qpidd 12365 qpidd mem REG 8,1 44992 786703 /lib/
libgcc_s-4.1.2-20080825.so.1
qpidd 12365 qpidd mem REG 8,1 1620400 789604 /lib/
i686/nosegneg/libc-2.5.so
qpidd 12365 qpidd mem REG 8,1 127720 786500 /lib/
i686/nosegneg/libpthread-2.5.so
qpidd 12365 qpidd mem REG 8,1 74616 786483 /lib/
libresolv-2.5.so
qpidd 12365 qpidd mem REG 8,1 43544 786447 /lib/
libcrypt-2.5.so
qpidd 12365 qpidd mem REG 8,1 2632 1213919 /usr/
lib/libaio.so.1.0.1
qpidd 12365 qpidd mem REG 8,1 15584 1247077 /usr/
lib/openais/libcpg.so.2.0.0
qpidd 12365 qpidd mem REG 8,1 14144 786738 /lib/
libuuid.so.1.2
qpidd 12365 qpidd mem REG 8,1 16068 1215646 /usr/
lib/libcman.so.2.0.115
qpidd 12365 qpidd mem REG 8,1 42068 786501 /lib/
i686/nosegneg/librt-2.5.so
qpidd 12365 qpidd mem REG 8,1 62328 1215614 /usr/
lib/libboost_filesystem.so.1.33.1
qpidd 12365 qpidd mem REG 8,1 124432 786503 /lib/
ld-2.5.so
qpidd 12365 qpidd mem REG 8,1 625546 1266311 /usr/
lib/qpid/daemon/replicating_listener.so
qpidd 12365 qpidd mem REG 8,1 213760 1215620 /usr/
lib/libboost_program_options.so.1.33.1
qpidd 12365 qpidd mem REG 8,1 1127841 1266315 /usr/
lib/qpid/daemon/msgstore.so
qpidd 12365 qpidd mem REG 8,1 946263 1215643 /usr/
lib/libsslcommon.so.0.1.0
qpidd 12365 qpidd mem REG 8,1 96924 1214052 /usr/
lib/libnssutil3.so
qpidd 12365 qpidd mem REG 8,1 925520 1214126 /usr/
lib/libstdc++.so.6.0.8
qpidd 12365 qpidd mem REG 8,1 1109436 1213951 /usr/
lib/libdb_cxx-4.3.so
qpidd 12365 qpidd mem REG 8,1 203340 1214124 /usr/
lib/libssl3.so
qpidd 12365 qpidd mem REG 8,1 839781 1266317 /usr/
lib/qpid/daemon/xml.so
qpidd 12365 qpidd mem REG 8,1 16636786 1215692 /usr/
lib/libqpidbroker.so.0.1.0
qpidd 12365 qpidd mem REG 8,1 226512 1214049 /usr/
lib/libnspr4.so
qpidd 12365 qpidd mem REG 8,1 6270148 1266314 /usr/
lib/qpid/daemon/cluster.so
qpidd 12365 qpidd mem REG 8,1 9912 1214097 /usr/
lib/libplds4.so
qpidd 12365 qpidd mem REG 8,1 6782769 1215639 /usr/
lib/libqpidclient.so.0.1.0
qpidd 12365 qpidd mem REG 8,1 587779 1261584 /usr/
lib/qpid/client/sslconnector.so
qpidd 12365 qpidd mem REG 8,1 14008 1214096 /usr/
lib/libplc4.so
qpidd 12365 qpidd mem REG 8,1 515215 1266312 /usr/
lib/qpid/daemon/replication_exchange.so
qpidd 12365 qpidd mem REG 8,1 558508 1266316 /usr/
lib/qpid/daemon/ssl.so
qpidd 12365 qpidd mem REG 8,1 1187124 1214050 /usr/
lib/libnss3.so
qpidd 12365 qpidd mem REG 8,1 4155068 1215591 /usr/
lib/libxerces-c.so.27.0
qpidd 12365 qpidd mem REG 8,1 3957632 1215596 /usr/
lib/libxqilla.so.3.0.0
qpidd 12365 qpidd 0u CHR 136,0 2 /dev/
pts/0
qpidd 12365 qpidd 1u CHR 136,0 2 /dev/
pts/0
qpidd 12365 qpidd 2u CHR 136,0 2 /dev/
pts/0
qpidd 12365 qpidd 3r FIFO 0,6 240193 pipe
qpidd 12365 qpidd 4w FIFO 0,6 240193 pipe
qpidd 12365 qpidd 5w REG 8,1 3556296254 98581 /var/
log/qpid/sfbroker.log
qpidd 12365 qpidd 6r CHR 1,9 1065 /dev/
urandom
qpidd 12365 qpidd 7r FIFO 0,6 240195 pipe
qpidd 12366 qpidd cwd DIR 8,1 4096 557058 /var/
lib/qpidd/.qpidd
qpidd 12366 qpidd rtd DIR 8,1 4096 2 /
qpidd 12366 qpidd txt REG 8,1 76165 1299146 /usr/
sbin/qpidd
qpidd 12366 qpidd mem REG 8,1 18031759 1215641 /usr/
lib/libqpidcommon.so.0.1.0
qpidd 12366 qpidd mem REG 8,1 14644 786449 /lib/
libdl-2.5.so
qpidd 12366 qpidd mem REG 8,1 97512 1214114 /usr/
lib/libsasl2.so.2.0.22
qpidd 12366 qpidd mem REG 8,1 206396 786499 /lib/
i686/nosegneg/libm-2.5.so
qpidd 12366 qpidd mem REG 8,1 44992 786703 /lib/
libgcc_s-4.1.2-20080825.so.1
qpidd 12366 qpidd mem REG 8,1 1620400 789604 /lib/
i686/nosegneg/libc-2.5.so
qpidd 12366 qpidd mem REG 8,1 127720 786500 /lib/
i686/nosegneg/libpthread-2.5.so
qpidd 12366 qpidd mem REG 8,1 74616 786483 /lib/
libresolv-2.5.so
qpidd 12366 qpidd mem REG 8,1 43544 786447 /lib/
libcrypt-2.5.so
qpidd 12366 qpidd mem REG 8,1 2632 1213919 /usr/
lib/libaio.so.1.0.1
qpidd 12366 qpidd mem REG 8,1 15584 1247077 /usr/
lib/openais/libcpg.so.2.0.0
qpidd 12366 qpidd mem REG 8,1 14144 786738 /lib/
libuuid.so.1.2
qpidd 12366 qpidd mem REG 8,1 16068 1215646 /usr/
lib/libcman.so.2.0.115
qpidd 12366 qpidd mem REG 8,1 42068 786501 /lib/
i686/nosegneg/librt-2.5.so
qpidd 12366 qpidd mem REG 8,1 62328 1215614 /usr/
lib/libboost_filesystem.so.1.33.1
qpidd 12366 qpidd mem REG 8,1 124432 786503 /lib/
ld-2.5.so
qpidd 12366 qpidd mem REG 8,1 625546 1266311 /usr/
lib/qpid/daemon/replicating_listener.so
qpidd 12366 qpidd mem REG 8,1 213760 1215620 /usr/
lib/libboost_program_options.so.1.33.1
qpidd 12366 qpidd mem REG 8,1 1127841 1266315 /usr/
lib/qpid/daemon/msgstore.so
qpidd 12366 qpidd mem REG 8,1 946263 1215643 /usr/
lib/libsslcommon.so.0.1.0
qpidd 12366 qpidd mem REG 8,1 96924 1214052 /usr/
lib/libnssutil3.so
qpidd 12366 qpidd mem REG 8,1 925520 1214126 /usr/
lib/libstdc++.so.6.0.8
qpidd 12366 qpidd mem REG 8,1 1109436 1213951 /usr/
lib/libdb_cxx-4.3.so
qpidd 12366 qpidd mem REG 8,1 203340 1214124 /usr/
lib/libssl3.so
qpidd 12366 qpidd mem REG 8,1 839781 1266317 /usr/
lib/qpid/daemon/xml.so
qpidd 12366 qpidd mem REG 8,1 16636786 1215692 /usr/
lib/libqpidbroker.so.0.1.0
qpidd 12366 qpidd mem REG 8,1 226512 1214049 /usr/
lib/libnspr4.so
qpidd 12366 qpidd mem REG 8,1 6270148 1266314 /usr/
lib/qpid/daemon/cluster.so
qpidd 12366 qpidd mem REG 8,1 9912 1214097 /usr/
lib/libplds4.so
qpidd 12366 qpidd mem REG 8,1 6782769 1215639 /usr/
lib/libqpidclient.so.0.1.0
qpidd 12366 qpidd mem REG 8,1 587779 1261584 /usr/
lib/qpid/client/sslconnector.so
qpidd 12366 qpidd mem REG 8,1 14008 1214096 /usr/
lib/libplc4.so
qpidd 12366 qpidd mem REG 8,1 515215 1266312 /usr/
lib/qpid/daemon/replication_exchange.so
qpidd 12366 qpidd mem REG 8,1 558508 1266316 /usr/
lib/qpid/daemon/ssl.so
qpidd 12366 qpidd mem REG 8,1 1187124 1214050 /usr/
lib/libnss3.so
qpidd 12366 qpidd mem REG 8,1 4155068 1215591 /usr/
lib/libxerces-c.so.27.0
qpidd 12366 qpidd mem REG 8,1 3957632 1215596 /usr/
lib/libxqilla.so.3.0.0
qpidd 12366 qpidd mem REG 8,96 16384 402653317 /vol/
qpidd/rhm/dat/__db.005
qpidd 12366 qpidd mem REG 8,96 450560 402653316 /vol/
qpidd/rhm/dat/__db.004
qpidd 12366 qpidd mem REG 8,96 294912 402653315 /vol/
qpidd/rhm/dat/__db.003
qpidd 12366 qpidd mem REG 8,96 278528 402653314 /vol/
qpidd/rhm/dat/__db.002
qpidd 12366 qpidd mem REG 8,96 24576 402653313 /vol/
qpidd/rhm/dat/__db.001
qpidd 12366 qpidd 0u CHR 1,3 778 /dev/
null
qpidd 12366 qpidd 1u CHR 1,3 778 /dev/
null
qpidd 12366 qpidd 2u CHR 1,3 778 /dev/
null
qpidd 12366 qpidd 3r FIFO 0,6 240193 pipe
qpidd 12366 qpidd 4w FIFO 0,6 240193 pipe
qpidd 12366 qpidd 5w REG 8,1 3556296254 98581 /var/
log/qpid/sfbroker.log
qpidd 12366 qpidd 6r CHR 1,9 1065 /dev/
urandom
qpidd 12366 qpidd 7r 0000 0,10 0 240197
eventpoll
qpidd 12366 qpidd 8w FIFO 0,6 240195 pipe
qpidd 12366 qpidd 9u unix 0xed2f5040 240196 socket
qpidd 12366 qpidd 10wW REG 8,96 0 131 /vol/
qpidd/lock
qpidd 12366 qpidd 11r FIFO 0,6 240198 pipe
qpidd 12366 qpidd 12w FIFO 0,6 240198 pipe
qpidd 12366 qpidd 13u REG 8,96 8192 402653319 /vol/
qpidd/rhm/dat/queues.db
qpidd 12366 qpidd 14u REG 8,96 8192 402653320 /vol/
qpidd/rhm/dat/config.db
qpidd 12366 qpidd 15u REG 8,96 8192 402653321 /vol/
qpidd/rhm/dat/exchanges.db
qpidd 12366 qpidd 16u REG 8,96 8192 402653322 /vol/
qpidd/rhm/dat/messages.db
qpidd 12366 qpidd 17u REG 8,96 8192 402653323 /vol/
qpidd/rhm/dat/mappings.db
qpidd 12366 qpidd 18u REG 8,96 8192 402653324 /vol/
qpidd/rhm/dat/bindings.db
qpidd 12366 qpidd 19u REG 8,96 8192 402653325 /vol/
qpidd/rhm/dat/general.db
qpidd 12366 qpidd 20u REG 8,96 10485760 402653318 /vol/
qpidd/rhm/dat/log.0000000001
qpidd 12366 qpidd 21r REG 8,96 1342177792 503316610 /vol/
qpidd/rhm/jrnl/000c/hits/JournalData.0001.jdat
On Mar 11, 2010, at 12:15 PM, Kim van der Riet wrote:
> A few questions:
>
> 1. What versions of the store and qpid are you using?
>
> 2. Are your messages persistent? Flow-to-disk will force transient
> messages to disk, but will discard them on recovery; persistent
> messages
> are written to disk anyway, but have their message content discarded
> when the policy limit is reached. I guess the latter is the case here.
>
> This behavior should be checked out... it looks suspicious.
>
> On Thu, 2010-03-11 at 11:01 -0800, Charles Woerner wrote:
>> A bit more information, at about the time Journal recovery phase 1 is
>> complete I see one of the two qpidd processes begin to acquire vast
>> amounts of memory (I assume this is the phase where it begins reading
>> up to max-queue-size of persistent queue data into memory). Once the
>> process acquires about 97% of available memory the system usually
>> dies
>> with the "critical Broker start-up failed: Cannot read from child
>> process.".
>>
>> I'm not surprised that the system dies when it runs out of memory.
>> But I hope someone can help me understand why qpidd is using so much
>> memory and whether I have my max-queue-size and store geometry
>> configured in a reasonable manner. I would hope that there isn't
>> some
>> inherent limitation where one can't recover more data from a
>> persistent journal than one has available memory.
>>
>> On Mar 10, 2010, at 10:55 PM, Charles Woerner wrote:
>>
>>> To follow up, it always fails at startup with this large queue,
>>> but I
>>> don't always see the "Broker startup failed: cannot read from child
>>> process" error. Sometimes it just dies.
>>>
>>
>> __
>>
>> Charles Woerner | cwoerner@demandbase.com | demandbase
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
__
Charles Woerner | cwoerner@demandbase.com | demandbase |
415.683.2669
Re: c++ broker startup problem with large journal containing lots
of data
Posted by Kim van der Riet <ki...@redhat.com>.
A few questions:
1. What versions of the store and qpid are you using?
2. Are your messages persistent? Flow-to-disk will force transient
messages to disk, but will discard them on recovery; persistent messages
are written to disk anyway, but have their message content discarded
when the policy limit is reached. I guess the latter is the case here.
This behavior should be checked out... it looks suspicious.
On Thu, 2010-03-11 at 11:01 -0800, Charles Woerner wrote:
> A bit more information, at about the time Journal recovery phase 1 is
> complete I see one of the two qpidd processes begin to acquire vast
> amounts of memory (I assume this is the phase where it begins reading
> up to max-queue-size of persistent queue data into memory). Once the
> process acquires about 97% of available memory the system usually dies
> with the "critical Broker start-up failed: Cannot read from child
> process.".
>
> I'm not surprised that the system dies when it runs out of memory.
> But I hope someone can help me understand why qpidd is using so much
> memory and whether I have my max-queue-size and store geometry
> configured in a reasonable manner. I would hope that there isn't some
> inherent limitation where one can't recover more data from a
> persistent journal than one has available memory.
>
> On Mar 10, 2010, at 10:55 PM, Charles Woerner wrote:
>
> > To follow up, it always fails at startup with this large queue, but I
> > don't always see the "Broker startup failed: cannot read from child
> > process" error. Sometimes it just dies.
> >
>
> __
>
> Charles Woerner | cwoerner@demandbase.com | demandbase
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: c++ broker startup problem with large journal containing lots of data
Posted by Charles Woerner <cw...@demandbase.com>.
A bit more information, at about the time Journal recovery phase 1 is
complete I see one of the two qpidd processes begin to acquire vast
amounts of memory (I assume this is the phase where it begins reading
up to max-queue-size of persistent queue data into memory). Once the
process acquires about 97% of available memory the system usually dies
with the "critical Broker start-up failed: Cannot read from child
process.".
I'm not surprised that the system dies when it runs out of memory.
But I hope someone can help me understand why qpidd is using so much
memory and whether I have my max-queue-size and store geometry
configured in a reasonable manner. I would hope that there isn't some
inherent limitation where one can't recover more data from a
persistent journal than one has available memory.
On Mar 10, 2010, at 10:55 PM, Charles Woerner wrote:
> To follow up, it always fails at startup with this large queue, but I
> don't always see the "Broker startup failed: cannot read from child
> process" error. Sometimes it just dies.
>
__
Charles Woerner | cwoerner@demandbase.com | demandbase
Re: size of the queue
Posted by Carl Trieloff <cc...@redhat.com>.
On 03/11/2010 02:22 AM, Arati Sethy wrote:
> Hello,
>
> May I know how to set "qpid.file_count"& "qpid.file_size" in declare queues as an argument for C++ broker.
> Thanks in advance.
>
>
> Regards,
> Arati
>
There is a helper class -- QueueOptions.h in the client headers.
Take a look at it
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
size of the queue
Posted by Arati Sethy <Ar...@tecnotree.com>.
Hello,
May I know how to set "qpid.file_count" & "qpid.file_size" in declare queues as an argument for C++ broker.
Thanks in advance.
Regards,
Arati
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: c++ broker startup problem with large journal containing lots of data
Posted by Charles Woerner <cw...@demandbase.com>.
To follow up, it always fails at startup with this large queue, but I
don't always see the "Broker startup failed: cannot read from child
process" error. Sometimes it just dies.
__
Charles Woerner | cwoerner@demandbase.com | demandbase |
415.683.2669