You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by ffrenchm <ff...@gmail.com> on 2009/02/23 16:45:34 UTC
QPID C++ Broker + persistent store
Hello again,
I'm currently using successfully QPid C++ Broker and BDB as message store. I
checkouted the qpid BDB module here :
http://anonsvn.jboss.org/repos/rhmessaging/store/trunk/cpp/
I've two questions :
1) I would like to know how I can disable the debug trace when I use the BDB
persisent store in QPid (I've to much traces like bellow and it's realy not
good for my benchmarks) :
2009-feb-23 16:34:10 trace RECV [127.0.0.1:60321]: Frame[BEbe; channel=0;
{SessionCompletedBody: commands={ [39936,39993] }; }]
2009-feb-23 16:34:10 debug fd058497-242f-4a1c-be96-425875bef247@guest@QPID:
sender marked completed: { [39936,39993] }
2009-feb-23 16:34:10 debug fd058497-242f-4a1c-be96-425875bef247@guest@QPID:
sender confirmed point moved to (39994+0)
2009-feb-23 16:34:10 trace RECV [127.0.0.1:60321]: Frame[BEbe; channel=0;
{MessageAcceptBody: transfers={ [39993,39993] }; }]
2009-feb-23 16:34:10 debug fd058497-242f-4a1c-be96-425875bef247@guest@QPID:
recv cmd 40047: {MessageAcceptBody: transfers={ [39993,39993] }; }
2009-feb-23 16:34:10 debug DeliveryRecord::setEnded() id=39993
2009-feb-23 16:34:10 debug Accepted 39993
2009-feb-23 16:34:10 debug fd058497-242f-4a1c-be96-425875bef247@guest@QPID:
receiver marked completed: 40047 incomplete: { } unknown-completed: {
[1,40047] }
2009-feb-23 16:34:10 trace RECV [127.0.0.1:60321]: Frame[BEbe; channel=0;
{SessionCompletedBody: commands={ [39936,39994] }; }]
2009-feb-23 16:34:10 debug fd058497-242f-4a1c-be96-425875bef247@guest@QPID:
sender marked completed: { [39936,39994] }
2009-feb-23 16:34:10 debug fd058497-242f-4a1c-be96-425875bef247@guest@QPID:
sender confirmed point moved to (39995+0)
2009-feb-23 16:34:10 trace RECV [127.0.0.1:60321]: Frame[BEbe; channel=0;
{MessageAcceptBody: transfers={ [39994,39994] }; }]
2009-feb-23 16:34:10 debug fd058497-242f-4a1c-be96-425875bef247@guest@QPID:
recv cmd 40048: {MessageAcceptBody: transfers={ [39994,39994] }; }
2009-feb-23 16:34:10 debug DeliveryRecord::setEnded() id=39994
2009-feb-23 16:34:10 debug Accepted 39994
2009-feb-23 16:34:10 debug fd058497-242f-4a1c-be96-425875bef247@guest@QPID:
receiver marked completed: 40048 incomplete: { } unknown-completed: {
[1,40048] }
2009-feb-23 16:34:10 trace RECV [127.0.0.1:60321]: Frame[BEbe; channel=0;
{SessionCompletedBody: commands={ [39936,39995] }; }]
2009-feb-23 16:34:10 debug fd058497-242f-4a1c-be96-425875bef247@guest@QPID:
sender marked completed: { [39936,39995] }
2009-feb-23 16:34:10 debug fd058497-242f-4a1c-be96-425875bef247@guest@QPID:
sender confirmed point moved to (39996+0)
2009-feb-23 16:34:10 trace RECV [127.0.0.1:60321]: Frame[BEbe; channel=0;
{MessageAcceptBody: transfers={ [39995,39995] }; }]
2009-feb-23 16:34:10 debug fd058497-242f-4a1c-be96-425875bef247@guest@QPID:
recv cmd 40049: {MessageAcceptBody: transfers={ [39995,39995] }; }
2009-feb-23 16:34:10 debug DeliveryRecord::setEnded() id=39995
2009-feb-23 16:34:10 debug Accepted 39995
2) When I want to get the help of the BDB module I get this :
Store Options:
--store-dir DIR Store directory location for persistence
(instead of using --data-dir value).
Required
if --no-data-dir is also used.
--num-jfiles N (8) Default number of files for each journal
instance (queue).
--jfile-size-pgs N (24) Default size for each journal file in
multiples
of read pages (1 read page = 64kiB)
--wcache-page-size N (32) Size of the pages in the write page cache in
KiB. Allowable values - powers of 2: 1, 2,
4,
... , 128. Lower values decrease latency at
the
expense of throughput.
--tpl-num-jfiles N (8) Number of files for transaction prepared
list
journal instance
--tpl-jfile-size-pgs N (24) Size of each transaction prepared list
journal
file in multiples of read pages (1 read page
=
64kiB)
--tpl-wcache-page-size N (4) Size of the pages in the transaction
prepared
list write page cache in KiB. Allowable
values
- powers of 2: 1, 2, 4, ... , 128. Lower
values
decrease latency at the expense of
throughput.
Where are the options to have synchronous / asynchronous IO ? Did I make a
mistake when downlowding the BDB module ? Is there another one ?
Thanks for all
--
View this message in context: http://n2.nabble.com/QPID-C%2B%2B-Broker-%2B-persistent-store-tp2372671p2372671.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by Gordon Sim <gs...@redhat.com>.
ffrenchm wrote:
> I'm currently working on a JMS client. I suppose that the equivalent of C++
> async session is a DUPS_OK_ACKNOWLEDGE session (I'm not sure) ...
Not really; DUPS_OK_ACKNOWLEDGE just controls the frequency with which
messages received by the client are acknowledged (allowing the broker to
dequeue them).
The async session in the C++ client means that all the commands invoked
on the session interface will return without waiting for the broker to
confirm that the command was processed (indeed it may well not even have
yet received the command). The client can then explicitly sync() at any
given point (or periodically flush() to request that the broker send
information about which commands it has completed, or wait for
completion etc).
There is a configuration option in the JMS client (sync_persistence =
true) to require that the send() method blocks for durable messages
(required for strict compliance with JMS, but obviously much slower).
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by ffrenchm <ff...@gmail.com>.
I'm currently working on a JMS client. I suppose that the equivalent of C++
async session is a DUPS_OK_ACKNOWLEDGE session (I'm not sure) ...
++
Carl Trieloff wrote:
>
>
> yes, for example in c++ there is a sync and async session. The async one
> return after calling and
> allows another message to be sent. Errors or handled async. sync session
> blocks on the call.
>
> The sync / flush commands can be used to control this yourself if you want
>
> This type of thing can be done from any client, which client do you care
> about?
> Carl.
>
>
> ffrenchm wrote:
>> Ok I understand your answer, but it raise me another question :) Is there
>> a
>> way to have a really asynchronous client/server communication when
>> putting
>> messages ? I mean the client send a message and then the server ack the
>> client without waiting that the message is on the disk ?
>>
>> Thanks ...
>>
>>
>> Carl Trieloff wrote:
>>
>>>
>>>> Well, I do not understand why the synchronous IO mode is no longer
>>>> supported. My experiences of queueing tell me that if you want really
>>>> safe
>>>> message store you need synchronous IO on message PUT. But maybe did I
>>>> miss
>>>> some new things which make this feature deprecated ?
>>>>
>>>>
>>>>
>>> This is a great question:
>>>
>>> The key is that the C++ broker is that it is an async process, so the
>>> broker has all the logic to know when all the async processing on a
>>> message is complete. So even with a async store, the broker will not ack
>>> the client until the message/update is physically written to disk. Given
>>> this there is no advantage to a sync mode. Async is faster in all cases
>>> - and all the exact same message guarantees are provided using async.
>>>
>>> The only reason the sync option was there was to enable testing while
>>> the async impl was been hardened.
>>>
>>> regards
>>> Carl.
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Apache Qpid - AMQP Messaging Implementation
>>> Project: http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>>>
>>>
>>>
>>
>>
>
>
>
--
View this message in context: http://n2.nabble.com/QPID-C%2B%2B-Broker-%2B-persistent-store-tp2372671p2373080.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by Carl Trieloff <cc...@redhat.com>.
yes, for example in c++ there is a sync and async session. The async one
return after calling and
allows another message to be sent. Errors or handled async. sync session
blocks on the call.
The sync / flush commands can be used to control this yourself if you want
This type of thing can be done from any client, which client do you care
about?
Carl.
ffrenchm wrote:
> Ok I understand your answer, but it raise me another question :) Is there a
> way to have a really asynchronous client/server communication when putting
> messages ? I mean the client send a message and then the server ack the
> client without waiting that the message is on the disk ?
>
> Thanks ...
>
>
> Carl Trieloff wrote:
>
>>
>>> Well, I do not understand why the synchronous IO mode is no longer
>>> supported. My experiences of queueing tell me that if you want really
>>> safe
>>> message store you need synchronous IO on message PUT. But maybe did I
>>> miss
>>> some new things which make this feature deprecated ?
>>>
>>>
>>>
>> This is a great question:
>>
>> The key is that the C++ broker is that it is an async process, so the
>> broker has all the logic to know when all the async processing on a
>> message is complete. So even with a async store, the broker will not ack
>> the client until the message/update is physically written to disk. Given
>> this there is no advantage to a sync mode. Async is faster in all cases
>> - and all the exact same message guarantees are provided using async.
>>
>> The only reason the sync option was there was to enable testing while
>> the async impl was been hardened.
>>
>> regards
>> Carl.
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>>
>>
>
>
Re: QPID C++ Broker + persistent store
Posted by ffrenchm <ff...@gmail.com>.
Ok I understand your answer, but it raise me another question :) Is there a
way to have a really asynchronous client/server communication when putting
messages ? I mean the client send a message and then the server ack the
client without waiting that the message is on the disk ?
Thanks ...
Carl Trieloff wrote:
>
>
>> Well, I do not understand why the synchronous IO mode is no longer
>> supported. My experiences of queueing tell me that if you want really
>> safe
>> message store you need synchronous IO on message PUT. But maybe did I
>> miss
>> some new things which make this feature deprecated ?
>>
>>
>
> This is a great question:
>
> The key is that the C++ broker is that it is an async process, so the
> broker has all the logic to know when all the async processing on a
> message is complete. So even with a async store, the broker will not ack
> the client until the message/update is physically written to disk. Given
> this there is no advantage to a sync mode. Async is faster in all cases
> - and all the exact same message guarantees are provided using async.
>
> The only reason the sync option was there was to enable testing while
> the async impl was been hardened.
>
> regards
> Carl.
>
>
>
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
>
--
View this message in context: http://n2.nabble.com/QPID-C%2B%2B-Broker-%2B-persistent-store-tp2372671p2372992.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by Carl Trieloff <cc...@redhat.com>.
> Well, I do not understand why the synchronous IO mode is no longer
> supported. My experiences of queueing tell me that if you want really safe
> message store you need synchronous IO on message PUT. But maybe did I miss
> some new things which make this feature deprecated ?
>
>
This is a great question:
The key is that the C++ broker is that it is an async process, so the
broker has all the logic to know when all the async processing on a
message is complete. So even with a async store, the broker will not ack
the client until the message/update is physically written to disk. Given
this there is no advantage to a sync mode. Async is faster in all cases
- and all the exact same message guarantees are provided using async.
The only reason the sync option was there was to enable testing while
the async impl was been hardened.
regards
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by Gordon Sim <gs...@redhat.com>.
ffrenchm wrote:
>
>
> Gordon Sim wrote:
>> ffrenchm wrote:
>>> Where are the options to have synchronous / asynchronous IO ?
>> That option is no longer supported. The store will always use
>> asynchronous IO for recording the enqueue/dequeue of messages (bdb is
>> really only used to record exchanges, bindings and other config).
>>
>>
>
> Well, I do not understand why the synchronous IO mode is no longer
> supported. My experiences of queueing tell me that if you want really safe
> message store you need synchronous IO on message PUT. But maybe did I miss
> some new things which make this feature deprecated ?
>
Asynchronous IO means that the write is not blocked until the data is on
disk, allowing the broker to start writing the next message immediately
and improving the throughput. However safety is still important as you
point out and the store tracks completion of all these asynchronous
writes and the broker will not acknowledge the receipt of the message to
the sending client until the store confirms that the message has reached
the disk.
I.e. once the broker tells the client it has completed the publication
request, the message is guaranteed to be on the disk.
The client can then independently vary the strategy used to request
information on completion of publications. It can send message
asynchronously, in which case the send method on the api will return
before the broker has fully processed the message (perhaps flushing
regularly to update its record of safe messages). Alternatively it can
request that the send method blocks until the message is confirmed by
the broker. (Or indeed some hybrid of the two).
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by ffrenchm <ff...@gmail.com>.
Gordon Sim wrote:
>
> ffrenchm wrote:
>> Hello again,
>>
>> I'm currently using successfully QPid C++ Broker and BDB as message
>> store. I
>> checkouted the qpid BDB module here :
>> http://anonsvn.jboss.org/repos/rhmessaging/store/trunk/cpp/
>>
>> I've two questions :
>>
>> 1) I would like to know how I can disable the debug trace when I use the
>> BDB
>> persisent store in QPid (I've to much traces like bellow and it's realy
>> not
>> good for my benchmarks) :
>
> Thats caused by a qpidd option and isn't directly related to the store.
> Are you specifying --trace (or -t) to qpidd? If not check whats in your
> environment or the default config file (qpidd --help should show the
> location of that).
>
>
Ooops, you're right I was specifying -t option to qpidd :)
Gordon Sim wrote:
>
>> 2) When I want to get the help of the BDB module I get this :
>>
>
> [...]
>
>> Where are the options to have synchronous / asynchronous IO ?
>
> That option is no longer supported. The store will always use
> asynchronous IO for recording the enqueue/dequeue of messages (bdb is
> really only used to record exchanges, bindings and other config).
>
>
Well, I do not understand why the synchronous IO mode is no longer
supported. My experiences of queueing tell me that if you want really safe
message store you need synchronous IO on message PUT. But maybe did I miss
some new things which make this feature deprecated ?
--
View this message in context: http://n2.nabble.com/QPID-C%2B%2B-Broker-%2B-persistent-store-tp2372671p2372777.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by Gordon Sim <gs...@redhat.com>.
ffrenchm wrote:
> Hello again,
>
> I'm currently using successfully QPid C++ Broker and BDB as message store. I
> checkouted the qpid BDB module here :
> http://anonsvn.jboss.org/repos/rhmessaging/store/trunk/cpp/
>
> I've two questions :
>
> 1) I would like to know how I can disable the debug trace when I use the BDB
> persisent store in QPid (I've to much traces like bellow and it's realy not
> good for my benchmarks) :
Thats caused by a qpidd option and isn't directly related to the store.
Are you specifying --trace (or -t) to qpidd? If not check whats in your
environment or the default config file (qpidd --help should show the
location of that).
> 2) When I want to get the help of the BDB module I get this :
>
[...]
> Where are the options to have synchronous / asynchronous IO ?
That option is no longer supported. The store will always use
asynchronous IO for recording the enqueue/dequeue of messages (bdb is
really only used to record exchanges, bindings and other config).
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Python and location of spec files (was Re: QPID C++ Broker +
persistent store)
Posted by "mffrench@axway.com" <mf...@axway.com>.
I previously tested with python 2.4 and I've the same problem without
your patch:
[mffrench@dekatonshir:~/work/QUEUERS/QPID/trunk/qpid/python]%./hello-world
Traceback (most recent call last):
File "./hello-world", line 21, in ?
from qpid.connection import Connection
File
"/home/mffrench/work/QUEUERS/QPID/trunk/qpid/python/qpid/__init__.py",
line 20, in ?
import spec, codec, connection, content, peer, delegate, client
File
"/home/mffrench/work/QUEUERS/QPID/trunk/qpid/python/qpid/spec.py", line
61, in ?
SPEC = load(default())
File
"/home/mffrench/work/QUEUERS/QPID/trunk/qpid/python/qpid/spec.py", line
52, in load
doc = mllib.xml_parse(specfile)
File
"/home/mffrench/work/QUEUERS/QPID/trunk/qpid/python/mllib/__init__.py",
line 59, in xml_parse
xml.sax.parse(source, p)
File "/usr/lib/python2.4/site-packages/_xmlplus/sax/__init__.py", line
31, in parse
parser.parse(filename_or_stream)
File "/usr/lib/python2.4/site-packages/_xmlplus/sax/expatreader.py",
line 109, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib/python2.4/site-packages/_xmlplus/sax/xmlreader.py",
line 123, in parse
self.feed(buffer)
File "/usr/lib/python2.4/site-packages/_xmlplus/sax/expatreader.py",
line 216, in feed
self._parser.Parse(data, isFinal)
File "/usr/lib/python2.4/site-packages/_xmlplus/sax/expatreader.py",
line 395, in external_entity_ref
self._source.getSystemId() or
File "/usr/lib/python2.4/site-packages/_xmlplus/sax/saxutils.py", line
523, in prepare_input_source
f = urllib2.urlopen(source.getSystemId())
File "/usr/lib/python2.4/urllib2.py", line 130, in urlopen
return _opener.open(url, data)
File "/usr/lib/python2.4/urllib2.py", line 350, in open
protocol = req.get_type()
File "/usr/lib/python2.4/urllib2.py", line 233, in get_type
raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type:
/home/mffrench/work/QUEUERS/QPID/trunk/qpid/specs/amqp.0-10.dtd
Gordon Sim wrote:
> ffrenchm wrote:
>> I have the same ...
>>
>> [mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%grep version
>> /usr/lib/python2.5/site-packages/_xmlplus/__init__.py
>> version_info = (0,8,4)
>> __version__ = "0.8.4"
>
> Ok, thanks for checking, must be something else then... Ted, Rafi, any
> ideas? I'm on python 2.4 still, but I believe we have 2.5 users who
> have not hit this issue previously?
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Python and location of spec files (was Re: QPID C++ Broker +
persistent store)
Posted by Gordon Sim <gs...@redhat.com>.
ffrenchm wrote:
> I have the same ...
>
> [mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%grep version
> /usr/lib/python2.5/site-packages/_xmlplus/__init__.py
> version_info = (0,8,4)
> __version__ = "0.8.4"
Ok, thanks for checking, must be something else then... Ted, Rafi, any
ideas? I'm on python 2.4 still, but I believe we have 2.5 users who have
not hit this issue previously?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Python and location of spec files (was Re: QPID C++ Broker +
persistent store)
Posted by ffrenchm <ff...@gmail.com>.
I have the same ...
[mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%grep version
/usr/lib/python2.5/site-packages/_xmlplus/__init__.py
version_info = (0,8,4)
__version__ = "0.8.4"
Gordon Sim wrote:
>
> Gordon Sim wrote:
>> I'm guessing the version of your xml processing libraries is different
>> from those tested on, and the processing of the dtd is somehow different.
>
> What version of pyxml do you have? (grep version
> /usr/lib/python2.5/site-packages/_xmlplus/__init__.py)
>
> I have 0.8.4
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
>
--
View this message in context: http://n2.nabble.com/QPID-C%2B%2B-Broker-%2B-persistent-store-tp2372671p2383332.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Python and location of spec files (was Re: QPID C++ Broker +
persistent store)
Posted by Gordon Sim <gs...@redhat.com>.
Gordon Sim wrote:
> I'm guessing the version of your xml processing libraries is different
> from those tested on, and the processing of the dtd is somehow different.
What version of pyxml do you have? (grep version
/usr/lib/python2.5/site-packages/_xmlplus/__init__.py)
I have 0.8.4
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Python and location of spec files (was Re: QPID C++ Broker +
persistent store)
Posted by ffrenchm <ff...@gmail.com>.
with your patch I can make work the hello_world successfully :)
Gordon Sim wrote:
>
> ffrenchm wrote:
>> I tested the packaged python client
>> (http://www.apache.org/dist/qpid/M4/qpid-python-M4.tar.gz) and I've the
>> same
>> problem (see bellow). Is there any python version prerequisites (mine is
>> 2.5).
>
> Sorry for the earlier incorrect diagnosis! As a temporary workaround you
> could try the attached patch to set the dtd to be a valid file:// url.
> I'm guessing the version of your xml processing libraries is different
> from those tested on, and the processing of the dtd is somehow different.
>
> Anyone have any insight as to what might be going on here? Can we turn
> off the loading of the dtd easily?
>
>>
>> [mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%echo $PYTHONPATH
>> /home/mffrench/work/QUEUERS/QPID/qpid-M4/python
>> [mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%ls
>> hello-world python specs
>> [mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%ls specs/
>> amqp.0-10.dtd amqp.0-10-qpid-errata.xml amqp.0-8.xml
>> amqp-dtx-preview.0-9.xml amqp-nogen.0-9.xml LICENSE
>> NOTICE
>> amqp.0-10-preview.xml amqp.0-10.xml amqp.0-9.xml
>> amqp-errata.0-9.xml cluster.0-8.xml management-schema.xml
>> [mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%./hello-world
>> Traceback (most recent call last):
>> File "./hello-world", line 36, in <module>
>> conn = Connection(connect(broker, port))
>> File
>> "/home/mffrench/work/QUEUERS/QPID/qpid-M4/python/qpid/connection.py",
>> line 69, in __init__
>> spec = load(default())
>> File "/home/mffrench/work/QUEUERS/QPID/qpid-M4/python/qpid/spec010.py",
>> line 679, in load
>> doc = mllib.xml_parse(xml)
>> File
>> "/home/mffrench/work/QUEUERS/QPID/qpid-M4/python/mllib/__init__.py",
>> line 59, in xml_parse
>> xml.sax.parse(source, p)
>> File "/usr/lib/python2.5/site-packages/_xmlplus/sax/__init__.py", line
>> 31,
>> in parse
>> parser.parse(filename_or_stream)
>> File "/usr/lib/python2.5/site-packages/_xmlplus/sax/expatreader.py",
>> line
>> 109, in parse
>> xmlreader.IncrementalParser.parse(self, source)
>> File "/usr/lib/python2.5/site-packages/_xmlplus/sax/xmlreader.py", line
>> 123, in parse
>> self.feed(buffer)
>> File "/usr/lib/python2.5/site-packages/_xmlplus/sax/expatreader.py",
>> line
>> 216, in feed
>> self._parser.Parse(data, isFinal)
>> File "/usr/lib/python2.5/site-packages/_xmlplus/sax/expatreader.py",
>> line
>> 396, in external_entity_ref
>> "")
>> File "/usr/lib/python2.5/site-packages/_xmlplus/sax/saxutils.py", line
>> 523, in prepare_input_source
>> f = urllib2.urlopen(source.getSystemId())
>> File "/usr/lib/python2.5/urllib2.py", line 124, in urlopen
>> return _opener.open(url, data)
>> File "/usr/lib/python2.5/urllib2.py", line 373, in open
>> protocol = req.get_type()
>> File "/usr/lib/python2.5/urllib2.py", line 244, in get_type
>> raise ValueError, "unknown url type: %s" % self.__original
>> ValueError: unknown url type:
>> /home/mffrench/work/QUEUERS/QPID/qpid-M4/specs/amqp.0-10.dtd
>>
>>
>>
>> ffrenchm wrote:
>>> hello,
>>>
>>> I already checkouted the specs folder under trunk/qpid :
>>> [mffrench@dekatonshir:~/work/QUEUERS/QPID/trunk/qpid]%ls -al
>>> /home/mffrench/work/QUEUERS/QPID/trunk/qpid/specs/
>>> total 1284
>>> drwxr-xr-x 3 mffrench mffrench 4096 2009-02-17 18:18 .
>>> drwxr-xr-x 15 mffrench mffrench 4096 2009-02-17 18:18 ..
>>> -rw-r--r-- 1 mffrench mffrench 9383 2009-02-17 18:18 amqp.0-10.dtd
>>> -rw-r--r-- 1 mffrench mffrench 280203 2009-02-17 18:18
>>> amqp.0-10-preview.xml
>>> -rw-r--r-- 1 mffrench mffrench 256524 2009-02-17 18:18
>>> amqp.0-10-qpid-errata.xml
>>> -rw-r--r-- 1 mffrench mffrench 256419 2009-02-17 18:18 amqp.0-10.xml
>>> -rw-r--r-- 1 mffrench mffrench 149480 2009-02-17 18:18 amqp.0-8.xml
>>> -rw-r--r-- 1 mffrench mffrench 214161 2009-02-17 18:18 amqp.0-9.xml
>>> -rw-r--r-- 1 mffrench mffrench 41935 2009-02-17 18:18
>>> amqp-dtx-preview.0-9.xml
>>> -rw-r--r-- 1 mffrench mffrench 1749 2009-02-17 18:18
>>> amqp-errata.0-9.xml
>>> -rw-r--r-- 1 mffrench mffrench 1420 2009-02-17 18:18
>>> amqp-nogen.0-9.xml
>>> -rw-r--r-- 1 mffrench mffrench 1764 2009-02-17 18:18 cluster.0-8.xml
>>> -rw-r--r-- 1 mffrench mffrench 18564 2009-02-17 18:18 LICENSE
>>> -rw-r--r-- 1 mffrench mffrench 18158 2009-02-17 18:18
>>> management-schema.xml
>>> -rw-r--r-- 1 mffrench mffrench 287 2009-02-17 18:18 NOTICE
>>> drwxr-xr-x 6 mffrench mffrench 4096 2009-02-23 13:16 .svn
>>>
>>> ++
>>>
>>>
>>>
>>> Gordon Sim wrote:
>>>> ffrenchm wrote:
>>>>> raise ValueError, "unknown url type: %s" % self.__original
>>>>> ValueError: unknown url type:
>>>>> /home/mffrench/work/QUEUERS/QPID/trunk/qpid/specs/amqp.0-10.dtd
>>>>>
>>>>> I'm working on Ubuntu 7.10 system. Do you have any idea what I can do
>>>>> to
>>>>> make python tools working ?
>>>> I think you just need to checkout the specs folder under /trunk/qpid
>>>> (or
>>>> at least amqp.0-10-qpid-errata.xml and amqp.0-10.dtd which contain the
>>>> 0-10 spec).
>>>>
>>>> ---------------------------------------------------------------------
>>>> Apache Qpid - AMQP Messaging Implementation
>>>> Project: http://qpid.apache.org
>>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>>
>>>>
>>>>
>>>
>>
>
>
> Index: specs/amqp.0-10-qpid-errata.xml
> ===================================================================
> --- specs/amqp.0-10-qpid-errata.xml (revision 747695)
> +++ specs/amqp.0-10-qpid-errata.xml (working copy)
> @@ -121,7 +121,7 @@
>
> -->
>
> -<!DOCTYPE amqp SYSTEM "amqp.0-10.dtd">
> +<!DOCTYPE amqp SYSTEM
> "file:///home/mffrench/work/QUEUERS/QPID/qpid-M4/specs/amqp.0-10.dtd">
>
> <amqp xmlns="http://www.amqp.org/schema/amqp.xsd"
> major="0" minor="10" port="5672">
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
--
View this message in context: http://n2.nabble.com/QPID-C%2B%2B-Broker-%2B-persistent-store-tp2372671p2383355.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Python and location of spec files (was Re: QPID C++ Broker +
persistent store)
Posted by Gordon Sim <gs...@redhat.com>.
ffrenchm wrote:
> I tested the packaged python client
> (http://www.apache.org/dist/qpid/M4/qpid-python-M4.tar.gz) and I've the same
> problem (see bellow). Is there any python version prerequisites (mine is
> 2.5).
Sorry for the earlier incorrect diagnosis! As a temporary workaround you
could try the attached patch to set the dtd to be a valid file:// url.
I'm guessing the version of your xml processing libraries is different
from those tested on, and the processing of the dtd is somehow different.
Anyone have any insight as to what might be going on here? Can we turn
off the loading of the dtd easily?
>
> [mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%echo $PYTHONPATH
> /home/mffrench/work/QUEUERS/QPID/qpid-M4/python
> [mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%ls
> hello-world python specs
> [mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%ls specs/
> amqp.0-10.dtd amqp.0-10-qpid-errata.xml amqp.0-8.xml
> amqp-dtx-preview.0-9.xml amqp-nogen.0-9.xml LICENSE NOTICE
> amqp.0-10-preview.xml amqp.0-10.xml amqp.0-9.xml
> amqp-errata.0-9.xml cluster.0-8.xml management-schema.xml
> [mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%./hello-world
> Traceback (most recent call last):
> File "./hello-world", line 36, in <module>
> conn = Connection(connect(broker, port))
> File "/home/mffrench/work/QUEUERS/QPID/qpid-M4/python/qpid/connection.py",
> line 69, in __init__
> spec = load(default())
> File "/home/mffrench/work/QUEUERS/QPID/qpid-M4/python/qpid/spec010.py",
> line 679, in load
> doc = mllib.xml_parse(xml)
> File "/home/mffrench/work/QUEUERS/QPID/qpid-M4/python/mllib/__init__.py",
> line 59, in xml_parse
> xml.sax.parse(source, p)
> File "/usr/lib/python2.5/site-packages/_xmlplus/sax/__init__.py", line 31,
> in parse
> parser.parse(filename_or_stream)
> File "/usr/lib/python2.5/site-packages/_xmlplus/sax/expatreader.py", line
> 109, in parse
> xmlreader.IncrementalParser.parse(self, source)
> File "/usr/lib/python2.5/site-packages/_xmlplus/sax/xmlreader.py", line
> 123, in parse
> self.feed(buffer)
> File "/usr/lib/python2.5/site-packages/_xmlplus/sax/expatreader.py", line
> 216, in feed
> self._parser.Parse(data, isFinal)
> File "/usr/lib/python2.5/site-packages/_xmlplus/sax/expatreader.py", line
> 396, in external_entity_ref
> "")
> File "/usr/lib/python2.5/site-packages/_xmlplus/sax/saxutils.py", line
> 523, in prepare_input_source
> f = urllib2.urlopen(source.getSystemId())
> File "/usr/lib/python2.5/urllib2.py", line 124, in urlopen
> return _opener.open(url, data)
> File "/usr/lib/python2.5/urllib2.py", line 373, in open
> protocol = req.get_type()
> File "/usr/lib/python2.5/urllib2.py", line 244, in get_type
> raise ValueError, "unknown url type: %s" % self.__original
> ValueError: unknown url type:
> /home/mffrench/work/QUEUERS/QPID/qpid-M4/specs/amqp.0-10.dtd
>
>
>
> ffrenchm wrote:
>> hello,
>>
>> I already checkouted the specs folder under trunk/qpid :
>> [mffrench@dekatonshir:~/work/QUEUERS/QPID/trunk/qpid]%ls -al
>> /home/mffrench/work/QUEUERS/QPID/trunk/qpid/specs/
>> total 1284
>> drwxr-xr-x 3 mffrench mffrench 4096 2009-02-17 18:18 .
>> drwxr-xr-x 15 mffrench mffrench 4096 2009-02-17 18:18 ..
>> -rw-r--r-- 1 mffrench mffrench 9383 2009-02-17 18:18 amqp.0-10.dtd
>> -rw-r--r-- 1 mffrench mffrench 280203 2009-02-17 18:18
>> amqp.0-10-preview.xml
>> -rw-r--r-- 1 mffrench mffrench 256524 2009-02-17 18:18
>> amqp.0-10-qpid-errata.xml
>> -rw-r--r-- 1 mffrench mffrench 256419 2009-02-17 18:18 amqp.0-10.xml
>> -rw-r--r-- 1 mffrench mffrench 149480 2009-02-17 18:18 amqp.0-8.xml
>> -rw-r--r-- 1 mffrench mffrench 214161 2009-02-17 18:18 amqp.0-9.xml
>> -rw-r--r-- 1 mffrench mffrench 41935 2009-02-17 18:18
>> amqp-dtx-preview.0-9.xml
>> -rw-r--r-- 1 mffrench mffrench 1749 2009-02-17 18:18
>> amqp-errata.0-9.xml
>> -rw-r--r-- 1 mffrench mffrench 1420 2009-02-17 18:18 amqp-nogen.0-9.xml
>> -rw-r--r-- 1 mffrench mffrench 1764 2009-02-17 18:18 cluster.0-8.xml
>> -rw-r--r-- 1 mffrench mffrench 18564 2009-02-17 18:18 LICENSE
>> -rw-r--r-- 1 mffrench mffrench 18158 2009-02-17 18:18
>> management-schema.xml
>> -rw-r--r-- 1 mffrench mffrench 287 2009-02-17 18:18 NOTICE
>> drwxr-xr-x 6 mffrench mffrench 4096 2009-02-23 13:16 .svn
>>
>> ++
>>
>>
>>
>> Gordon Sim wrote:
>>> ffrenchm wrote:
>>>> raise ValueError, "unknown url type: %s" % self.__original
>>>> ValueError: unknown url type:
>>>> /home/mffrench/work/QUEUERS/QPID/trunk/qpid/specs/amqp.0-10.dtd
>>>>
>>>> I'm working on Ubuntu 7.10 system. Do you have any idea what I can do to
>>>> make python tools working ?
>>> I think you just need to checkout the specs folder under /trunk/qpid (or
>>> at least amqp.0-10-qpid-errata.xml and amqp.0-10.dtd which contain the
>>> 0-10 spec).
>>>
>>> ---------------------------------------------------------------------
>>> Apache Qpid - AMQP Messaging Implementation
>>> Project: http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>>>
>>>
>>
>
Re: Python and location of spec files (was Re: QPID C++ Broker +
persistent store)
Posted by ffrenchm <ff...@gmail.com>.
I tested the packaged python client
(http://www.apache.org/dist/qpid/M4/qpid-python-M4.tar.gz) and I've the same
problem (see bellow). Is there any python version prerequisites (mine is
2.5).
[mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%echo $PYTHONPATH
/home/mffrench/work/QUEUERS/QPID/qpid-M4/python
[mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%ls
hello-world python specs
[mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%ls specs/
amqp.0-10.dtd amqp.0-10-qpid-errata.xml amqp.0-8.xml
amqp-dtx-preview.0-9.xml amqp-nogen.0-9.xml LICENSE NOTICE
amqp.0-10-preview.xml amqp.0-10.xml amqp.0-9.xml
amqp-errata.0-9.xml cluster.0-8.xml management-schema.xml
[mffrench@dekatonshir:~/work/QUEUERS/QPID/qpid-M4]%./hello-world
Traceback (most recent call last):
File "./hello-world", line 36, in <module>
conn = Connection(connect(broker, port))
File "/home/mffrench/work/QUEUERS/QPID/qpid-M4/python/qpid/connection.py",
line 69, in __init__
spec = load(default())
File "/home/mffrench/work/QUEUERS/QPID/qpid-M4/python/qpid/spec010.py",
line 679, in load
doc = mllib.xml_parse(xml)
File "/home/mffrench/work/QUEUERS/QPID/qpid-M4/python/mllib/__init__.py",
line 59, in xml_parse
xml.sax.parse(source, p)
File "/usr/lib/python2.5/site-packages/_xmlplus/sax/__init__.py", line 31,
in parse
parser.parse(filename_or_stream)
File "/usr/lib/python2.5/site-packages/_xmlplus/sax/expatreader.py", line
109, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib/python2.5/site-packages/_xmlplus/sax/xmlreader.py", line
123, in parse
self.feed(buffer)
File "/usr/lib/python2.5/site-packages/_xmlplus/sax/expatreader.py", line
216, in feed
self._parser.Parse(data, isFinal)
File "/usr/lib/python2.5/site-packages/_xmlplus/sax/expatreader.py", line
396, in external_entity_ref
"")
File "/usr/lib/python2.5/site-packages/_xmlplus/sax/saxutils.py", line
523, in prepare_input_source
f = urllib2.urlopen(source.getSystemId())
File "/usr/lib/python2.5/urllib2.py", line 124, in urlopen
return _opener.open(url, data)
File "/usr/lib/python2.5/urllib2.py", line 373, in open
protocol = req.get_type()
File "/usr/lib/python2.5/urllib2.py", line 244, in get_type
raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type:
/home/mffrench/work/QUEUERS/QPID/qpid-M4/specs/amqp.0-10.dtd
ffrenchm wrote:
>
> hello,
>
> I already checkouted the specs folder under trunk/qpid :
> [mffrench@dekatonshir:~/work/QUEUERS/QPID/trunk/qpid]%ls -al
> /home/mffrench/work/QUEUERS/QPID/trunk/qpid/specs/
> total 1284
> drwxr-xr-x 3 mffrench mffrench 4096 2009-02-17 18:18 .
> drwxr-xr-x 15 mffrench mffrench 4096 2009-02-17 18:18 ..
> -rw-r--r-- 1 mffrench mffrench 9383 2009-02-17 18:18 amqp.0-10.dtd
> -rw-r--r-- 1 mffrench mffrench 280203 2009-02-17 18:18
> amqp.0-10-preview.xml
> -rw-r--r-- 1 mffrench mffrench 256524 2009-02-17 18:18
> amqp.0-10-qpid-errata.xml
> -rw-r--r-- 1 mffrench mffrench 256419 2009-02-17 18:18 amqp.0-10.xml
> -rw-r--r-- 1 mffrench mffrench 149480 2009-02-17 18:18 amqp.0-8.xml
> -rw-r--r-- 1 mffrench mffrench 214161 2009-02-17 18:18 amqp.0-9.xml
> -rw-r--r-- 1 mffrench mffrench 41935 2009-02-17 18:18
> amqp-dtx-preview.0-9.xml
> -rw-r--r-- 1 mffrench mffrench 1749 2009-02-17 18:18
> amqp-errata.0-9.xml
> -rw-r--r-- 1 mffrench mffrench 1420 2009-02-17 18:18 amqp-nogen.0-9.xml
> -rw-r--r-- 1 mffrench mffrench 1764 2009-02-17 18:18 cluster.0-8.xml
> -rw-r--r-- 1 mffrench mffrench 18564 2009-02-17 18:18 LICENSE
> -rw-r--r-- 1 mffrench mffrench 18158 2009-02-17 18:18
> management-schema.xml
> -rw-r--r-- 1 mffrench mffrench 287 2009-02-17 18:18 NOTICE
> drwxr-xr-x 6 mffrench mffrench 4096 2009-02-23 13:16 .svn
>
> ++
>
>
>
> Gordon Sim wrote:
>>
>> ffrenchm wrote:
>>> raise ValueError, "unknown url type: %s" % self.__original
>>> ValueError: unknown url type:
>>> /home/mffrench/work/QUEUERS/QPID/trunk/qpid/specs/amqp.0-10.dtd
>>>
>>> I'm working on Ubuntu 7.10 system. Do you have any idea what I can do to
>>> make python tools working ?
>>
>> I think you just need to checkout the specs folder under /trunk/qpid (or
>> at least amqp.0-10-qpid-errata.xml and amqp.0-10.dtd which contain the
>> 0-10 spec).
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>>
>
>
--
View this message in context: http://n2.nabble.com/QPID-C%2B%2B-Broker-%2B-persistent-store-tp2372671p2383097.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Python and location of spec files (was Re: QPID C++ Broker +
persistent store)
Posted by ffrenchm <ff...@gmail.com>.
hello,
I already checkouted the specs folder under trunk/qpid :
[mffrench@dekatonshir:~/work/QUEUERS/QPID/trunk/qpid]%ls -al
/home/mffrench/work/QUEUERS/QPID/trunk/qpid/specs/
total 1284
drwxr-xr-x 3 mffrench mffrench 4096 2009-02-17 18:18 .
drwxr-xr-x 15 mffrench mffrench 4096 2009-02-17 18:18 ..
-rw-r--r-- 1 mffrench mffrench 9383 2009-02-17 18:18 amqp.0-10.dtd
-rw-r--r-- 1 mffrench mffrench 280203 2009-02-17 18:18
amqp.0-10-preview.xml
-rw-r--r-- 1 mffrench mffrench 256524 2009-02-17 18:18
amqp.0-10-qpid-errata.xml
-rw-r--r-- 1 mffrench mffrench 256419 2009-02-17 18:18 amqp.0-10.xml
-rw-r--r-- 1 mffrench mffrench 149480 2009-02-17 18:18 amqp.0-8.xml
-rw-r--r-- 1 mffrench mffrench 214161 2009-02-17 18:18 amqp.0-9.xml
-rw-r--r-- 1 mffrench mffrench 41935 2009-02-17 18:18
amqp-dtx-preview.0-9.xml
-rw-r--r-- 1 mffrench mffrench 1749 2009-02-17 18:18 amqp-errata.0-9.xml
-rw-r--r-- 1 mffrench mffrench 1420 2009-02-17 18:18 amqp-nogen.0-9.xml
-rw-r--r-- 1 mffrench mffrench 1764 2009-02-17 18:18 cluster.0-8.xml
-rw-r--r-- 1 mffrench mffrench 18564 2009-02-17 18:18 LICENSE
-rw-r--r-- 1 mffrench mffrench 18158 2009-02-17 18:18
management-schema.xml
-rw-r--r-- 1 mffrench mffrench 287 2009-02-17 18:18 NOTICE
drwxr-xr-x 6 mffrench mffrench 4096 2009-02-23 13:16 .svn
++
Gordon Sim wrote:
>
> ffrenchm wrote:
>> raise ValueError, "unknown url type: %s" % self.__original
>> ValueError: unknown url type:
>> /home/mffrench/work/QUEUERS/QPID/trunk/qpid/specs/amqp.0-10.dtd
>>
>> I'm working on Ubuntu 7.10 system. Do you have any idea what I can do to
>> make python tools working ?
>
> I think you just need to checkout the specs folder under /trunk/qpid (or
> at least amqp.0-10-qpid-errata.xml and amqp.0-10.dtd which contain the
> 0-10 spec).
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
>
--
View this message in context: http://n2.nabble.com/QPID-C%2B%2B-Broker-%2B-persistent-store-tp2372671p2382723.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Python and location of spec files (was Re: QPID C++ Broker +
persistent store)
Posted by Gordon Sim <gs...@redhat.com>.
ffrenchm wrote:
> raise ValueError, "unknown url type: %s" % self.__original
> ValueError: unknown url type:
> /home/mffrench/work/QUEUERS/QPID/trunk/qpid/specs/amqp.0-10.dtd
>
> I'm working on Ubuntu 7.10 system. Do you have any idea what I can do to
> make python tools working ?
I think you just need to checkout the specs folder under /trunk/qpid (or
at least amqp.0-10-qpid-errata.xml and amqp.0-10.dtd which contain the
0-10 spec).
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by ffrenchm <ff...@gmail.com>.
Very good answer for another question :) I know the qpid python tools since
the begining but when trying to run the hello-world tools I got this error :
[mffrench@dekatonshir:~/work/QUEUERS/QPID/trunk/qpid/python]%echo
$PYTHONPATH
/home/mffrench/work/QUEUERS/QPID/trunk/qpid/python
[mffrench@dekatonshir:~/work/QUEUERS/QPID/trunk/qpid/python]%./hello-world
Traceback (most recent call last):
File "./hello-world", line 21, in <module>
from qpid.connection import Connection
File
"/home/mffrench/work/QUEUERS/QPID/trunk/qpid/python/qpid/__init__.py", line
20, in <module>
import spec, codec, connection, content, peer, delegate, client
File "/home/mffrench/work/QUEUERS/QPID/trunk/qpid/python/qpid/spec.py",
line 61, in <module>
SPEC = load(default())
File "/home/mffrench/work/QUEUERS/QPID/trunk/qpid/python/qpid/spec.py",
line 52, in load
doc = mllib.xml_parse(specfile)
File
"/home/mffrench/work/QUEUERS/QPID/trunk/qpid/python/mllib/__init__.py", line
59, in xml_parse
xml.sax.parse(source, p)
File "/usr/lib/python2.5/site-packages/_xmlplus/sax/__init__.py", line 31,
in parse
parser.parse(filename_or_stream)
File "/usr/lib/python2.5/site-packages/_xmlplus/sax/expatreader.py", line
109, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib/python2.5/site-packages/_xmlplus/sax/xmlreader.py", line
123, in parse
self.feed(buffer)
File "/usr/lib/python2.5/site-packages/_xmlplus/sax/expatreader.py", line
216, in feed
self._parser.Parse(data, isFinal)
File "/usr/lib/python2.5/site-packages/_xmlplus/sax/expatreader.py", line
396, in external_entity_ref
"")
File "/usr/lib/python2.5/site-packages/_xmlplus/sax/saxutils.py", line
523, in prepare_input_source
f = urllib2.urlopen(source.getSystemId())
File "/usr/lib/python2.5/urllib2.py", line 124, in urlopen
return _opener.open(url, data)
File "/usr/lib/python2.5/urllib2.py", line 373, in open
protocol = req.get_type()
File "/usr/lib/python2.5/urllib2.py", line 244, in get_type
raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type:
/home/mffrench/work/QUEUERS/QPID/trunk/qpid/specs/amqp.0-10.dtd
I'm working on Ubuntu 7.10 system. Do you have any idea what I can do to
make python tools working ?
Thanks for all
Gordon Sim wrote:
>
> ffrenchm wrote:
>> Is there any way to force the flow-to-disk option for all messages which
>> are
>> putted in this queue ?
>
> One option is to create the queue with specific policy details using the
> qpid-config tool.
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
>
--
View this message in context: http://n2.nabble.com/QPID-C%2B%2B-Broker-%2B-persistent-store-tp2372671p2373391.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by Gordon Sim <gs...@redhat.com>.
ffrenchm wrote:
> Is there any way to force the flow-to-disk option for all messages which are
> putted in this queue ?
One option is to create the queue with specific policy details using the
qpid-config tool.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by Gordon Sim <gs...@redhat.com>.
Kim van der Riet wrote:
> On Mon, 2009-02-23 at 09:24 -0800, ffrenchm wrote:
>> Is there any way to force the flow-to-disk option for all messages which are
>> putted in this queue ?
>>
>
> Perhaps a terminology misunderstanding?
>
> Messages which are important and which must be recovered in the event of
> a broker failure must be marked durable, and be sent to a queue which
> has also been set to durable. These messages are always written to disk
> before they may be consumed, and will be recovered in the event of a
> broker failure. This is the normal mode to use for reliable messaging.
>
> Flow-to-disk is used to send messages to disk which are transient (ie do
> NOT need to be written to disk for reliability) when broker resources
> are running low.
We use flow-to-disk somewhat loosely/incorrectly perhaps.
Currently where the queue policy is set to flow-to-disk (only available
for durable queues) then the content of enqueued messages that would
take the queue over the preconfigured limit (either in terms of the
number of messages or their aggregate size), are released from memory
and will be read from disk when they are delivered to clients.
This happens for either durable messages (which are ordinarily written
to disk anyway) or transient messages (which are not).
As Kim points out though, the performance as perceived by subscribers in
both cases will currently drop considerably if this behaviour is hit so
it is primarily useful as a last resort to prevent excessive use of
memory. For best performance you want to set the limit high enough to
avoid this behaviour during normal operation.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by Kim van der Riet <ki...@redhat.com>.
On Mon, 2009-02-23 at 14:45 -0500, Kim van der Riet wrote:
> Flow-to-disk is used to send messages to disk which are transient (ie do
> NOT need to be written to disk for reliability) when broker resources
> are running low. If you have lots of transient messages waiting to be
> consumed, then at some point, these messages can also be written to
> disk, and the memory freed. When those messages are consumed, their
> content must first be read from the disk to restore the memory content,
> then sent to the consumer. These flow-to-disk messages are NOT recovered
> in the event of a failure. Flow-to-disk is not recommended for regular
> operation of the broker as it incurs a significant performance penalty.
> It is intended to be used as a last resort to manage a possible low
> memory condition.
>
> Flow-to-disk is controlled by setting broker limits on the cumulative
> size of messages on a queue (--default-queue-limit) and/or the message
> size of any single message (--staging-threshold). Note that flow-to-disk
> works only on durable queues.
>
I must clarify one point:
Messages which are large (greater than the --staging-threshold limit)
can also be durable. All large messages are currently stored in BDB
externally to the async journal, and if they are durable will be
restored along with all other durable messages on recovery. Transient
(non-durable) messages will be discarded on recovery, even though they
were written to disk for memory/space reasons.
Kim
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by Kim van der Riet <ki...@redhat.com>.
On Mon, 2009-02-23 at 09:24 -0800, ffrenchm wrote:
> Is there any way to force the flow-to-disk option for all messages which are
> putted in this queue ?
>
Perhaps a terminology misunderstanding?
Messages which are important and which must be recovered in the event of
a broker failure must be marked durable, and be sent to a queue which
has also been set to durable. These messages are always written to disk
before they may be consumed, and will be recovered in the event of a
broker failure. This is the normal mode to use for reliable messaging.
Flow-to-disk is used to send messages to disk which are transient (ie do
NOT need to be written to disk for reliability) when broker resources
are running low. If you have lots of transient messages waiting to be
consumed, then at some point, these messages can also be written to
disk, and the memory freed. When those messages are consumed, their
content must first be read from the disk to restore the memory content,
then sent to the consumer. These flow-to-disk messages are NOT recovered
in the event of a failure. Flow-to-disk is not recommended for regular
operation of the broker as it incurs a significant performance penalty.
It is intended to be used as a last resort to manage a possible low
memory condition.
Flow-to-disk is controlled by setting broker limits on the cumulative
size of messages on a queue (--default-queue-limit) and/or the message
size of any single message (--staging-threshold). Note that flow-to-disk
works only on durable queues.
I'm not sure why you would want to force flow-to-disk for all messages
unless you mean that they should all be durable.
>
> ffrenchm wrote:
> >
> >
> >
> > Kim van der Riet wrote:
> >>
> >>
> >> This store is only asynchronous. Although BDB is still used, it is no
> >> longer on the message path for normal usage patterns - ie no message
> >> data is stored in BDB. However, BDB will be used if flow-to-disk is
> >> triggered, either because the messages are too large or other conditions
> >> that may trigger this condition. All message content and transactions
> >> are no stored in the async journal.
> >>
> >> I hope this helps,
> >> Kim
> >>
> >>
> >
> > Thanks Kim, another question for you : I'm currently using JMS to create
> > queue 'test-queue' (in fact I reuse your bench tools). I already read this
> > page http://qpid.apache.org/cheat-sheet-for-configuring-queue-options.html
> > which explains how we can set the flow to disk option for a queue. But
> > there is only C++ API sample. Is there any way to define the queue options
> > thanks JMS API ? If not I've maybe an idea ;)
> >
> > ++
> >
>
Sorry, my knowledge of JMS is non-existent, I have no idea on this
question.
Kim
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by ffrenchm <ff...@gmail.com>.
Is there any way to force the flow-to-disk option for all messages which are
putted in this queue ?
++
ffrenchm wrote:
>
>
>
> Kim van der Riet wrote:
>>
>>
>> This store is only asynchronous. Although BDB is still used, it is no
>> longer on the message path for normal usage patterns - ie no message
>> data is stored in BDB. However, BDB will be used if flow-to-disk is
>> triggered, either because the messages are too large or other conditions
>> that may trigger this condition. All message content and transactions
>> are no stored in the async journal.
>>
>> I hope this helps,
>> Kim
>>
>>
>
> Thanks Kim, another question for you : I'm currently using JMS to create
> queue 'test-queue' (in fact I reuse your bench tools). I already read this
> page http://qpid.apache.org/cheat-sheet-for-configuring-queue-options.html
> which explains how we can set the flow to disk option for a queue. But
> there is only C++ API sample. Is there any way to define the queue options
> thanks JMS API ? If not I've maybe an idea ;)
>
> ++
>
--
View this message in context: http://n2.nabble.com/QPID-C%2B%2B-Broker-%2B-persistent-store-tp2372671p2373209.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by ffrenchm <ff...@gmail.com>.
Kim van der Riet wrote:
>
>
> This store is only asynchronous. Although BDB is still used, it is no
> longer on the message path for normal usage patterns - ie no message
> data is stored in BDB. However, BDB will be used if flow-to-disk is
> triggered, either because the messages are too large or other conditions
> that may trigger this condition. All message content and transactions
> are no stored in the async journal.
>
> I hope this helps,
> Kim
>
>
Thanks Kim, another question for you : I'm currently using JMS to create
queue 'test-queue' (in fact I reuse your bench tools). I already read this
page http://qpid.apache.org/cheat-sheet-for-configuring-queue-options.html
which explains how we can set the flow to disk option for a queue. But there
is only C++ API sample. Is there any way to define the queue options thanks
JMS API ? If not I've maybe an idea ;)
++
--
View this message in context: http://n2.nabble.com/QPID-C%2B%2B-Broker-%2B-persistent-store-tp2372671p2372975.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID C++ Broker + persistent store
Posted by Kim van der Riet <ki...@redhat.com>.
On Mon, 2009-02-23 at 07:45 -0800, ffrenchm wrote:
> I've two questions :
>
> 1) I would like to know how I can disable the debug trace when I use the BDB
> persisent store in QPid (I've to much traces like bellow and it's realy not
> good for my benchmarks) :
>
> 2009-feb-23 16:34:10 trace RECV [127.0.0.1:60321]: Frame[BEbe; channel=0;
> {SessionCompletedBody: commands={ [39936,39993] }; }]
...
> 2009-feb-23 16:34:10 debug DeliveryRecord::setEnded() id=39995
> 2009-feb-23 16:34:10 debug Accepted 39995
These traces are controlled by the broker options -t/--trace or
--log-enable; the store has no separate controls for traces. If you use
-t, then the trace is turned ON for the broker AND all loaded modules.
Instead of that, you can use the --log-enable option, using trace+,
debug+ info+, notice+ warning+, error+ or critical+ as arguments for the
different trace levels. The "+" is optional, it prints all messages
levels at the one specified and above (eg warning prints only warning
messages; warning+ prints warning, error and critical messages.) The
-t/--trace option is equivalent to --log-enable trace+. See the broker
help info under --log-enable for details.
> Where are the options to have synchronous / asynchronous IO ? Did I make a
> mistake when downlowding the BDB module ? Is there another one ?
This store is only asynchronous. Although BDB is still used, it is no
longer on the message path for normal usage patterns - ie no message
data is stored in BDB. However, BDB will be used if flow-to-disk is
triggered, either because the messages are too large or other conditions
that may trigger this condition. All message content and transactions
are no stored in the async journal.
I hope this helps,
Kim
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org