You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Fraser Adams <fr...@blueyonder.co.uk> on 2014/03/14 16:38:05 UTC
Proton's new async features - any examples? 'cause I'm scratching
my head!
Hey folks,
I've noticed that PROTON-525/531/534 cover work to expose some bits of
messenger that were previously internal and allow messenger to be driven
from an external poll/select/epoll.
I'm quite interested in this from the perspective of the JavaScript
bindings that I'm working on, but to be honest I'm currently left
scratching my head trying to figure out how the new APIs are intended to
work.
I don't suppose that there are any examples available?
I currently have a recv-async.c and send-async.c (attached) they are
still a bit hacky as they are currently work in progress as I push the
necessary features into emscripten (the C->JavaScript compiler I'm
using) but they both work in either native C or JavaScript (the
emscripten_set_network_callback gets triggered by WebSocket activity and
allows fully async behaviour, so I don't need any nasty polling).
I've just merged the latest proton-c stuff to the branch I'm working on
for the JavaScript bindings and everything is still working nicely with
the current approach, but I'm guessing that the new capabilities might
be able to make things more efficient?
I'm currently working on actual binding code, so I can call messenger
direct from native JavaScript as opposed to compiling C/C++ into
JavaScript, so far it has got a lot of parallels with the python
bindings - though clearly only async stuff makes any sense for JavaScript.
I'd really appreciate tips and code samples from the folks who have been
working on this API.
Cheers,
Frase
Re: Proton's new async features - any examples? 'cause I'm scratching
my head!
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Sun, Mar 16, 2014 at 8:47 AM, Fraser Adams <fraser.adams@blueyonder.co.uk
> wrote:
> On 16/03/14 10:21, Rafael Schloming wrote:
>
>>
>> and
>>>>
>>>>> Scanning dependencies of target docs-py
>>>>> +-----------------------------------------------------------
>>>>> ---------------
>>>>> | In /home/fadams/qpid/qpid-trunk/proton/proton-c/bindings/python/
>>>>> | proton.py:
>>>>> | Import failed (but source code parsing was successful).
>>>>> | Error: NameError: name 'PN_CONNECTION_STATE' is not defined (line
>>>>> | 3292)
>>>>> |
>>>>>
>>>>>
>>>>> It looks like your new API docs are created OK, but I've raised
>>>>> PROTON-535
>>>>> to cover it.
>>>>>
>>>>> Did you do a make prior to doing make docs or just do make docs? The
>>>>>
>>>> epydoc
>>>> generation imports the swig modules so it needs the C code to be built
>>>> in
>>>> order to run in its entirety, however it falls back to source only mode
>>>> if
>>>> the import fails.
>>>>
>>>> --Rafael
>>>>
>>>> I've just done
>>>>
>>> make clean
>>> make
>>> make docs
>>>
>>> and it does the same thing.
>>>
>>
>> That's interesting. I'm definitely not seeing that issue. What happens
>> when
>> you run python and type 'import proton' manually? Is it possible your
>> PYTHONPATH and/or LD_LIBRARY_PATH are pointing you to an older version or
>> something? Your epydoc version might also shed some light on the problem.
>> I'm on 3.0.1.
>>
>> --Rafael
>>
>> So this is weird:
>
> bringing up a terminal and in my home directory
>
> running python and then doing "import proton" directly into the python
> shell doesn't give any errors.
>
>
>
> Changing directory to
>
> <proton>/examples/messenger/py
>
> I can also happily run the send.py, recv.py, send_async.py, recv_async.py
> messenger examples with no errors by doing.
> ./recv.py
> etc. and also
> python recv.py
>
> and in that directory running python and then doing "import proton"
> directly into the python shell doesn't give any errors either.
>
>
>
>
> However, changing directory to
>
> <proton>/proton-c/bindings/python
>
> Then running python and then doing "import proton" directly into the
> python shell I see:
>
> >>> import proton
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "proton.py", line 3292, in <module>
> class Event:
> File "proton.py", line 3294, in Event
> CONNECTION_STATE = PN_CONNECTION_STATE
>
> NameError: name 'PN_CONNECTION_STATE' is not defined
>
>
>
> Hmmmm, I wonder. I've previously done a "make install" which was a little
> while ago using a 0.6 Proton, I did that to build Qpid with AMQP 1.0
> support, but I've not done it for a little while (I think that there are
> currently some incompatibilities with 0.7 so I've held off). At a guess the
> reason that I can just run ./recv.py in <proton>/examples/messenger/py and
> why import proton can actually see proton there and in my home directory is
> due to the *installed* version. But calling import python in
> <proton>/proton-c/bindings/python will see the version in that directory
> in preference to the installed version?
>
> So I'm wondering if there's perhaps an issue with proton.py on trunk??
>
So my guess is that when you cd into the bindings directory and import from
there you are ending up with a mix of the trunk python code importing your
older swigged module from its installed location, and looking for a
constant that wasn't defined in 0.6, but is now there in 0.7.
The safe way to work with trunk builds and ensure that you're not picking
up anything from local installs is to source the config.sh file that is in
the root of your source checkout. This shell script will set up your
LD_LIBRARY_PATH, your PYTHONPATH, etc so that you are always picking up
stuff from the source checkout. Note that the shell script assumes you
either use an in-tree build or a build directory named "build" if you do
something different you will need to set the CPROTON_BUILD environment
variable before sourcing config.sh.
--Rafael
Re: Proton's new async features - any examples? 'cause I'm scratching
my head!
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 16/03/14 10:21, Rafael Schloming wrote:
>
>>> and
>>>> Scanning dependencies of target docs-py
>>>> +-----------------------------------------------------------
>>>> ---------------
>>>> | In /home/fadams/qpid/qpid-trunk/proton/proton-c/bindings/python/
>>>> | proton.py:
>>>> | Import failed (but source code parsing was successful).
>>>> | Error: NameError: name 'PN_CONNECTION_STATE' is not defined (line
>>>> | 3292)
>>>> |
>>>>
>>>>
>>>> It looks like your new API docs are created OK, but I've raised
>>>> PROTON-535
>>>> to cover it.
>>>>
>>>> Did you do a make prior to doing make docs or just do make docs? The
>>> epydoc
>>> generation imports the swig modules so it needs the C code to be built in
>>> order to run in its entirety, however it falls back to source only mode if
>>> the import fails.
>>>
>>> --Rafael
>>>
>>> I've just done
>> make clean
>> make
>> make docs
>>
>> and it does the same thing.
>
> That's interesting. I'm definitely not seeing that issue. What happens when
> you run python and type 'import proton' manually? Is it possible your
> PYTHONPATH and/or LD_LIBRARY_PATH are pointing you to an older version or
> something? Your epydoc version might also shed some light on the problem.
> I'm on 3.0.1.
>
> --Rafael
>
So this is weird:
bringing up a terminal and in my home directory
running python and then doing "import proton" directly into the python
shell doesn't give any errors.
Changing directory to
<proton>/examples/messenger/py
I can also happily run the send.py, recv.py, send_async.py,
recv_async.py messenger examples with no errors by doing.
./recv.py
etc. and also
python recv.py
and in that directory running python and then doing "import proton"
directly into the python shell doesn't give any errors either.
However, changing directory to
<proton>/proton-c/bindings/python
Then running python and then doing "import proton" directly into the
python shell I see:
>>> import proton
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "proton.py", line 3292, in <module>
class Event:
File "proton.py", line 3294, in Event
CONNECTION_STATE = PN_CONNECTION_STATE
NameError: name 'PN_CONNECTION_STATE' is not defined
Hmmmm, I wonder. I've previously done a "make install" which was a
little while ago using a 0.6 Proton, I did that to build Qpid with AMQP
1.0 support, but I've not done it for a little while (I think that there
are currently some incompatibilities with 0.7 so I've held off). At a
guess the reason that I can just run ./recv.py in
<proton>/examples/messenger/py and why import proton can actually see
proton there and in my home directory is due to the *installed* version.
But calling import python in <proton>/proton-c/bindings/python will see
the version in that directory in preference to the installed version?
So I'm wondering if there's perhaps an issue with proton.py on trunk??
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Proton's new async features - any examples? 'cause I'm scratching
my head!
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Sat, Mar 15, 2014 at 4:10 PM, Fraser Adams <fraser.adams@blueyonder.co.uk
> wrote:
> Thanks for your responses to the async questions - I've responded to the
> doxygen ones below.
>
> Cheers,
> Frase
>
>
>
>>
>>> FYI when I tried "make docs" it mostly looks like it works but I saw
>>> several warnings:
>>>
>>> warning: ignoring unsupported tag `INLINE_SIMPLE_STRUCTS =' at line 313,
>>> file user.doxygen
>>> warning: ignoring unsupported tag `CITE_BIB_FILES =' at line 583,
>>> file user.doxygen
>>> warning: ignoring unsupported tag `MATHJAX_EXTENSIONS =' at line
>>> 1166,
>>> file user.doxygen
>>> warning: ignoring unsupported tag `LATEX_BIB_STYLE =' at line
>>> 1285,
>>> file user.doxygen
>>> warning: ignoring unsupported tag `INTERACTIVE_SVG =' at line
>>> 1699,
>>> file user.doxygen
>>>
>>>
>>> Hmm, I don't seem to see these. What version of doxygen are you using?
>>
> Ah, it's a fairly old version - 1.7.4
Ok, I'm on version 1.8.3.1, so that probably explains that. I don't know if
there is a way to parametrize the options in those files or if it is worth
doing so. I'm not really a doxygen expert, so far I've just figured out
enough about it to make it do what I want.
>
>
>
>
>> and
>>>
>>> Scanning dependencies of target docs-py
>>> +-----------------------------------------------------------
>>> ---------------
>>> | In /home/fadams/qpid/qpid-trunk/proton/proton-c/bindings/python/
>>> | proton.py:
>>> | Import failed (but source code parsing was successful).
>>> | Error: NameError: name 'PN_CONNECTION_STATE' is not defined (line
>>> | 3292)
>>> |
>>>
>>>
>>> It looks like your new API docs are created OK, but I've raised
>>> PROTON-535
>>> to cover it.
>>>
>>> Did you do a make prior to doing make docs or just do make docs? The
>> epydoc
>> generation imports the swig modules so it needs the C code to be built in
>> order to run in its entirety, however it falls back to source only mode if
>> the import fails.
>>
>> --Rafael
>>
>> I've just done
>
> make clean
> make
> make docs
>
> and it does the same thing.
That's interesting. I'm definitely not seeing that issue. What happens when
you run python and type 'import proton' manually? Is it possible your
PYTHONPATH and/or LD_LIBRARY_PATH are pointing you to an older version or
something? Your epydoc version might also shed some light on the problem.
I'm on 3.0.1.
--Rafael
Re: Proton's new async features - any examples? 'cause I'm scratching
my head!
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Thanks for your responses to the async questions - I've responded to the
doxygen ones below.
Cheers,
Frase
>
>>
>> FYI when I tried "make docs" it mostly looks like it works but I saw
>> several warnings:
>>
>> warning: ignoring unsupported tag `INLINE_SIMPLE_STRUCTS =' at line 313,
>> file user.doxygen
>> warning: ignoring unsupported tag `CITE_BIB_FILES =' at line 583,
>> file user.doxygen
>> warning: ignoring unsupported tag `MATHJAX_EXTENSIONS =' at line 1166,
>> file user.doxygen
>> warning: ignoring unsupported tag `LATEX_BIB_STYLE =' at line 1285,
>> file user.doxygen
>> warning: ignoring unsupported tag `INTERACTIVE_SVG =' at line 1699,
>> file user.doxygen
>>
>>
> Hmm, I don't seem to see these. What version of doxygen are you using?
Ah, it's a fairly old version - 1.7.4
>
>> and
>>
>> Scanning dependencies of target docs-py
>> +-----------------------------------------------------------
>> ---------------
>> | In /home/fadams/qpid/qpid-trunk/proton/proton-c/bindings/python/
>> | proton.py:
>> | Import failed (but source code parsing was successful).
>> | Error: NameError: name 'PN_CONNECTION_STATE' is not defined (line
>> | 3292)
>> |
>>
>>
>> It looks like your new API docs are created OK, but I've raised PROTON-535
>> to cover it.
>>
> Did you do a make prior to doing make docs or just do make docs? The epydoc
> generation imports the swig modules so it needs the C code to be built in
> order to run in its entirety, however it falls back to source only mode if
> the import fails.
>
> --Rafael
>
I've just done
make clean
make
make docs
and it does the same thing.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Proton's new async features - any examples? 'cause I'm scratching
my head!
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Sat, Mar 15, 2014 at 5:38 AM, Fraser Adams <fraser.adams@blueyonder.co.uk
> wrote:
> Thanks Rafael,
> I'll take a look at this.
>
> TBH I had actually done a bit of grepping around and had come across
> SelectableMessengerTest, but unfortunately I think that only confused me
> more :-/
>
> I guess that my mental model of how I'd expect to use this would be in a
> notifier loop analogous to a "broken apart" version of:
>
> while (1) {
> pn_messenger_work(messenger, -1); // Block indefinitely until there
> has been socket activity.
> process();
> }
>
> So seeing the non-blocking select and the way Pump was being used in a
> "busy wait" loop messed with my head - though I suspect that was just down
> to me "over analysing" things :-D
>
Ah, yes. The "busy wait" as you call it is actually just finishing up all
pending I/O operations. The loop terminates as soon as there is no more I/O
to perform. This works because the test is sequentially scripting both
sides of the socket interaction, so there is never any need to wait around
for a concurrent thread/process to finish sending you data, and therefore
no point in having a timeout that is > 0. If the body of pump_once were to
use a non zero timeout and be put in an infinite loop then you basically
have a "normal" event loop (minus the deadline part).
> I have to say that I agree with the comments raised by Darryl Pierce
> yesterday evening about some of the names being a little confusing.
>
> Out of curiosity how much more efficient would you expect this approach to
> be? I'm guessing if you have several messenger instances on the go and a
> high message rate it's likely to make quite a difference and you can also
> use epoll now too I guess.
>
I actually wouldn't expect it to make all that much of a difference
performance-wise since messenger's internal wait loop is implemented using
the same selectables API that it exposes. In fact for a language like
python it might be marginally faster to leave the loop in C rather than
reimplementing it in python. The API is really more about flexibility with
how you embed messenger into your environment. If you have an existing I/O
loop then this allows you to use it. The only case I can think of where
this would make a difference performance-wise is if you wanted to have
1000's of messengers in a single process without having a thread per
messenger.
As for select vs poll vs epoll, the internal wait loop uses poll, however
the performance difference between select, poll, and epoll really depends
not only on the total number of connections you have, but on how your I/O
activity is distributed. The case where epoll really shines is if you have
a lot of idle connections and a few active ones. This is because the poll
interface forces you to loop over all the fds to figure out which ones have
events, whereas epoll just gives you back the events directly. If a high
proportion of your fds frequently have events though, then epoll's
advantage diminishes. In fact I've seen comparisons where poll comes out
faster between the two.
>
>
> FYI when I tried "make docs" it mostly looks like it works but I saw
> several warnings:
>
> warning: ignoring unsupported tag `INLINE_SIMPLE_STRUCTS =' at line 313,
> file user.doxygen
> warning: ignoring unsupported tag `CITE_BIB_FILES =' at line 583,
> file user.doxygen
> warning: ignoring unsupported tag `MATHJAX_EXTENSIONS =' at line 1166,
> file user.doxygen
> warning: ignoring unsupported tag `LATEX_BIB_STYLE =' at line 1285,
> file user.doxygen
> warning: ignoring unsupported tag `INTERACTIVE_SVG =' at line 1699,
> file user.doxygen
>
>
Hmm, I don't seem to see these. What version of doxygen are you using?
>
> and
>
> Scanning dependencies of target docs-py
> +-----------------------------------------------------------
> ---------------
> | In /home/fadams/qpid/qpid-trunk/proton/proton-c/bindings/python/
> | proton.py:
> | Import failed (but source code parsing was successful).
> | Error: NameError: name 'PN_CONNECTION_STATE' is not defined (line
> | 3292)
> |
>
>
> It looks like your new API docs are created OK, but I've raised PROTON-535
> to cover it.
>
Did you do a make prior to doing make docs or just do make docs? The epydoc
generation imports the swig modules so it needs the C code to be built in
order to run in its entirety, however it falls back to source only mode if
the import fails.
--Rafael
Re: Proton's new async features - any examples? 'cause I'm scratching
my head!
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Thanks Rafael,
I'll take a look at this.
TBH I had actually done a bit of grepping around and had come across
SelectableMessengerTest, but unfortunately I think that only confused me
more :-/
I guess that my mental model of how I'd expect to use this would be in a
notifier loop analogous to a "broken apart" version of:
while (1) {
pn_messenger_work(messenger, -1); // Block indefinitely until there
has been socket activity.
process();
}
So seeing the non-blocking select and the way Pump was being used in a
"busy wait" loop messed with my head - though I suspect that was just
down to me "over analysing" things :-D
I have to say that I agree with the comments raised by Darryl Pierce
yesterday evening about some of the names being a little confusing.
Out of curiosity how much more efficient would you expect this approach
to be? I'm guessing if you have several messenger instances on the go
and a high message rate it's likely to make quite a difference and you
can also use epoll now too I guess.
FYI when I tried "make docs" it mostly looks like it works but I saw
several warnings:
warning: ignoring unsupported tag `INLINE_SIMPLE_STRUCTS =' at line
313, file user.doxygen
warning: ignoring unsupported tag `CITE_BIB_FILES =' at line
583, file user.doxygen
warning: ignoring unsupported tag `MATHJAX_EXTENSIONS =' at line
1166, file user.doxygen
warning: ignoring unsupported tag `LATEX_BIB_STYLE =' at line
1285, file user.doxygen
warning: ignoring unsupported tag `INTERACTIVE_SVG =' at line
1699, file user.doxygen
and
Scanning dependencies of target docs-py
+--------------------------------------------------------------------------
| In /home/fadams/qpid/qpid-trunk/proton/proton-c/bindings/python/
| proton.py:
| Import failed (but source code parsing was successful).
| Error: NameError: name 'PN_CONNECTION_STATE' is not defined (line
| 3292)
|
It looks like your new API docs are created OK, but I've raised
PROTON-535 to cover it.
Best Regards,
Frase
On 14/03/14 20:01, Rafael Schloming wrote:
> I don't have examples yet, other than the SelectableMessengerTest in
> messenger.py. I did just commit the doxygen API-docs for all that stuff a
> couple of minutes ago, so you might start there. You'll need to do a 'make
> docs' from a trunk proton checkout and then point your browser to
> <your-build-directory>/proton-c/docs/api/html.
>
> Have a read through of what's there now and let me know what questions you
> still have. I'm happy to explain more, but I'd like to get a sense for what
> gaps there are left in the docs.
>
> --Rafael
>
>
> On Fri, Mar 14, 2014 at 11:38 AM, Fraser Adams <
> fraser.adams@blueyonder.co.uk> wrote:
>
>> Hey folks,
>> I've noticed that PROTON-525/531/534 cover work to expose some bits of
>> messenger that were previously internal and allow messenger to be driven
>> from an external poll/select/epoll.
>>
>> I'm quite interested in this from the perspective of the JavaScript
>> bindings that I'm working on, but to be honest I'm currently left
>> scratching my head trying to figure out how the new APIs are intended to
>> work.
>>
>> I don't suppose that there are any examples available?
>>
>>
>> I currently have a recv-async.c and send-async.c (attached) they are still
>> a bit hacky as they are currently work in progress as I push the necessary
>> features into emscripten (the C->JavaScript compiler I'm using) but they
>> both work in either native C or JavaScript (the emscripten_set_network_callback
>> gets triggered by WebSocket activity and allows fully async behaviour, so I
>> don't need any nasty polling).
>>
>> I've just merged the latest proton-c stuff to the branch I'm working on
>> for the JavaScript bindings and everything is still working nicely with the
>> current approach, but I'm guessing that the new capabilities might be able
>> to make things more efficient?
>>
>> I'm currently working on actual binding code, so I can call messenger
>> direct from native JavaScript as opposed to compiling C/C++ into
>> JavaScript, so far it has got a lot of parallels with the python bindings -
>> though clearly only async stuff makes any sense for JavaScript.
>>
>>
>> I'd really appreciate tips and code samples from the folks who have been
>> working on this API.
>>
>> Cheers,
>> Frase
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Proton's new async features - any examples? 'cause I'm scratching
my head!
Posted by Rafael Schloming <rh...@alum.mit.edu>.
I don't have examples yet, other than the SelectableMessengerTest in
messenger.py. I did just commit the doxygen API-docs for all that stuff a
couple of minutes ago, so you might start there. You'll need to do a 'make
docs' from a trunk proton checkout and then point your browser to
<your-build-directory>/proton-c/docs/api/html.
Have a read through of what's there now and let me know what questions you
still have. I'm happy to explain more, but I'd like to get a sense for what
gaps there are left in the docs.
--Rafael
On Fri, Mar 14, 2014 at 11:38 AM, Fraser Adams <
fraser.adams@blueyonder.co.uk> wrote:
> Hey folks,
> I've noticed that PROTON-525/531/534 cover work to expose some bits of
> messenger that were previously internal and allow messenger to be driven
> from an external poll/select/epoll.
>
> I'm quite interested in this from the perspective of the JavaScript
> bindings that I'm working on, but to be honest I'm currently left
> scratching my head trying to figure out how the new APIs are intended to
> work.
>
> I don't suppose that there are any examples available?
>
>
> I currently have a recv-async.c and send-async.c (attached) they are still
> a bit hacky as they are currently work in progress as I push the necessary
> features into emscripten (the C->JavaScript compiler I'm using) but they
> both work in either native C or JavaScript (the emscripten_set_network_callback
> gets triggered by WebSocket activity and allows fully async behaviour, so I
> don't need any nasty polling).
>
> I've just merged the latest proton-c stuff to the branch I'm working on
> for the JavaScript bindings and everything is still working nicely with the
> current approach, but I'm guessing that the new capabilities might be able
> to make things more efficient?
>
> I'm currently working on actual binding code, so I can call messenger
> direct from native JavaScript as opposed to compiling C/C++ into
> JavaScript, so far it has got a lot of parallels with the python bindings -
> though clearly only async stuff makes any sense for JavaScript.
>
>
> I'd really appreciate tips and code samples from the folks who have been
> working on this API.
>
> Cheers,
> Frase
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>