You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Gordon Sim <gs...@redhat.com> on 2013/06/21 12:25:11 UTC

Federation on Windows (was Re: Qpid post-mortem and request for suggestions for (my) next release challenge (10M msgs/sec on Windows))

On 06/17/2013 07:31 PM, Steve Huston wrote:
>>> And anything you can think of for dynamically load balancing across
>> brokers?
>>
>> Honestly, I think the simplest solution overall is for us to get federation
>> working on windows. I assume its some issue in the IO layer.
>> Does anyone have a concrete understanding of what the problem is and
>> what is required to fix it?
>>
>> Any volunteers from our windows experts to take a look (Cliff, Chuck,
>> Andrew, Steve)?
>
> https://issues.apache.org/jira/browse/QPID-2199
>
> It is likely not a very complicated issue once it's understood more. Getting the time/funding to do it has been the only impediment.

A view was expressed that this was because the AsynchConnector for 
windows was not in fact asynchronous. I'm a little sceptical of that, 
though I admit I haven't looked into the issue at all.

The underlying socket is reported to be established. As Ted suggests in 
his comment, perhaps that the link registry has not been notified or was 
unable to match the notification to the pending link.

The connection id and federation link matching was very fragile in 
earlier versions (and seems now to be a little more robust). For example 
https://issues.apache.org/jira/browse/QPID-4315, where SSL federation 
broke if hostnames were used qith qpid-route.

Could a similar issue be (or have been) behind the windows problem?

Has anyone tried with a recent broker? E.g. 0.22 or trunk? There have 
been quite a few changes in the codepaths involved.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Re: Federation on Windows (was Re: Qpid post-mortem and request for suggestions for (my) next release challenge (10M msgs/sec on Windows))

Posted by Gordon Sim <gs...@redhat.com>.
On 06/21/2013 04:31 PM, Gordon Sim wrote:
> Federation as is provides a way of spreading connections across brokers
> for pub-sub style traffic. It doesn't yet support a federated queue
> (i.e. a queue which is split between different brokers abut for which
> each message is consumed by one consumer in the network).

Or putting it another way, it works reasonably well for balancing load 
between non-competing consumers, but doesn't do the coordination 
required for competing consumers.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Re: Federation on Windows (was Re: Qpid post-mortem and request for suggestions for (my) next release challenge (10M msgs/sec on Windows))

Posted by Gordon Sim <gs...@redhat.com>.
On 06/21/2013 04:23 PM, Kerry Bonin wrote:
> Is there a clear explanation anywhere of exactly what Broker federation
> offers without corosync?

Broker federation (which has never used corosync) is described here:

http://qpid.apache.org/books/0.20/AMQP-Messaging-Broker-CPP-Book/html/chap-Messaging_User_Guide-Broker_Federation.html

The new HA functionality (which replaces the previous cluster plugin 
which required direct corosync support) is described here:

http://qpid.apache.org/books/0.20/AMQP-Messaging-Broker-CPP-Book/html/chap-Messaging_User_Guide-Active_Passive_Cluster.html

> If I can meet my requirements by fixing this, and
> the scope of work is within an order of magnitude of rolling my own load
> balancing, I'd rather fix Qpid.

Federation as is provides a way of spreading connections across brokers 
for pub-sub style traffic. It doesn't yet support a federated queue 
(i.e. a queue which is split between different brokers abut for which 
each message is consumed by one consumer in the network).

>  I looked at it once, and it was in some
> overly clever and difficult to follow code, so I stopped, but I've spent a
> lot of time in the broker and low level Windows transport layers fixing
> other bugs, as has another senior member of my team...

If you haven't already done so I would strongly recommend a quick retest 
on 0.22 (or trunk) as there have been a lot of changes in the codepaths 
involved.

If not I'd be happy to try and help debugging in answering questions and 
suggesting things to check...



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Re: Federation on Windows (was Re: Qpid post-mortem and request for suggestions for (my) next release challenge (10M msgs/sec on Windows))

Posted by Kerry Bonin <ke...@gmail.com>.
Is there a clear explanation anywhere of exactly what Broker federation
offers without corosync?  If I can meet my requirements by fixing this, and
the scope of work is within an order of magnitude of rolling my own load
balancing, I'd rather fix Qpid.  I looked at it once, and it was in some
overly clever and difficult to follow code, so I stopped, but I've spent a
lot of time in the broker and low level Windows transport layers fixing
other bugs, as has another senior member of my team...


On Fri, Jun 21, 2013 at 5:25 AM, Gordon Sim <gs...@redhat.com> wrote:

> On 06/17/2013 07:31 PM, Steve Huston wrote:
>
>> And anything you can think of for dynamically load balancing across
>>>>
>>> brokers?
>>>
>>> Honestly, I think the simplest solution overall is for us to get
>>> federation
>>> working on windows. I assume its some issue in the IO layer.
>>> Does anyone have a concrete understanding of what the problem is and
>>> what is required to fix it?
>>>
>>> Any volunteers from our windows experts to take a look (Cliff, Chuck,
>>> Andrew, Steve)?
>>>
>>
>> https://issues.apache.org/**jira/browse/QPID-2199<https://issues.apache.org/jira/browse/QPID-2199>
>>
>> It is likely not a very complicated issue once it's understood more.
>> Getting the time/funding to do it has been the only impediment.
>>
>
> A view was expressed that this was because the AsynchConnector for windows
> was not in fact asynchronous. I'm a little sceptical of that, though I
> admit I haven't looked into the issue at all.
>
> The underlying socket is reported to be established. As Ted suggests in
> his comment, perhaps that the link registry has not been notified or was
> unable to match the notification to the pending link.
>
> The connection id and federation link matching was very fragile in earlier
> versions (and seems now to be a little more robust). For example
> https://issues.apache.org/**jira/browse/QPID-4315<https://issues.apache.org/jira/browse/QPID-4315>,
> where SSL federation broke if hostnames were used qith qpid-route.
>
> Could a similar issue be (or have been) behind the windows problem?
>
> Has anyone tried with a recent broker? E.g. 0.22 or trunk? There have been
> quite a few changes in the codepaths involved.
>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.**org<de...@qpid.apache.org>
> For additional commands, e-mail: dev-help@qpid.apache.org
>
>