Fwd: Apache.NMS and Apache.NMS.XMS

[Stéphane Ramet <st...@gmail.com>]

Transfered as a public archive for transparency.

Refers to JIRA issue AMQNET-185
<https://issues.apache.org/jira/browse/AMQNET-185>.

---------- Forwarded message ----------
From: Stéphane Ramet <st...@gmail.com>
Date: Sat, Dec 19, 2015 at 1:51 AM
Subject: Re: Apache.NMS and Apache.NMS.XMS
To: Jim Gomes <jg...@apache.org>


Hi Jim,

busy period indeed.

As I found it difficult to get a fair estimate of the amount of new code, I
thought joining the code to my reply would be the easiest way for you to
get an appreciation.

I attached 4 files :
1/ "2015-12-18_Apache.NMS.XMS.trunk.src.main.7z" is the main XMS provider
source code.
2/ "2015-12-18_Apache.NMS.XMS.trunk.src.test.7z" is the unit test source
code.
3/ "2015-12-18_Apache.EMS+MSMQ.QueueBrowser.7z" are patches to the EMS and
MSMQ providers for queue browsing capabilities.
4/ "2015-12-18_Apache.NMS.test.evidence.7z" is not really code but the
current state of test results, both NUnit and additional tests.

Getting into more details :

*1/ Main source code*
Most of the efforts in the main code went into the initialization of the
ConnectionFactory properties from the specified broker URI ("XMSEnum.cs",
"XMSConvert.cs", "ConnectionFactory.cs"), plus support for administered
object ("InitialContext.cs").
To document the URI attributes, and decouple their names from the object
properties, I introduced an "UriAttribute" annotation and rewrote the
"IntrospectionSupport.cs" to support that.


*2/ Unit test code*The unit test part would probably need more in-depth
discussions. I wanted to reuse most of the NMS unit test code, since this
covers a lot of standard use cases. But I didn't manage to use it as is,
for various reasons:

- in the NMS test code, most destinations get a random / guid-based name,
which implies that the provider should be able to dynamically create
destinations. It is not the case with XMS (neither the MSMQ provider as far
as I could see, and probably some others since JMS doesn't support it).
Therefore I needed a way to specify to the core NMS nunit code the queues
and topics it needed to address. I decided to pass references to nodes in
the configuration file. It still defaults to guid-based name generation if
a null reference is passed.

- some patterns in the NMS test code are not supported in XMS. For
instance, XMS does not support changing the ClientId after the Connection
is created; the ClientId must be set in the Factory beforehand. Since
Connection creation is done in the NMSTestSupport, which is the root for
all test modules, the logic cannot be easily changed from one provider to
the other. I decided to implement some kind of bridge pattern : I
introduced an NMSTest root, which forwards method calls to the "external",
overridable NMSTestSupport.

- to more easily adapt to my test environment, I introduced an environment
variable pointing to my specific "xmsprovider-test.config" file. I removed
the ReplaceEnvVar parts since I thought they were not needed any more, but
they could still easily be reintroduced if needed.

- I had to adapt some parts of the NMS unit test code to cope with another
limitation in XMS : it doesn't support both synchronous and asynchronous
operations in the same session. Therefore, in test cases where messages are
sent synchronously and received asynchronously, I introduced a second
session for consuming messages. I believe this is still representative of
real world implementations, where you would have separate sessions for
producing and consuming messages.

The final architecture is schematically :

Apache.NMS.XMS.Test.*Test
  --(inherits from)-->
    Apache.NMS.Test.*Test
      --(inherits from)-->
        Apache.NMS.Test.NMSTest
           --(invokes)-->
             Apache.NMS.Test.NMSTestSupport
                --(where required, overriden by)-->
                  Apache.XMS.Test.XMSTestSupport

The Apache.NMS.XMS.Test.*Test is a "thin" object responsible for :
- instanciating the XMSTestSupport object and passing it upwards to the
NMSTest through the constructors chain.
- invoking the relevant Apache.NMS.Test.*Test methods, passing queue or
topic references needed for running the test cases.

I hope I didn't go too far with this one. As I mentioned earlier, I'm new
to Open Source contribution. But as I understand it now, this would usually
be debated through discussion streams beforehand.

Plus the XMS unit test code now includes potentially standard / not
provider-specific code, which is not ideal.

I'll let you tell me how you would like this part to be processed.

*3/ QueueBrowser*
Nothing special to mention.


*4/ Current test status*
I couldn't get a hand on a running publish/subscribe service in my
customer's environment. Therefore I couldn't write and run ad-hoc tests for
topics as planned. And many NMS unit tests, as they rely on topics, fail as
well.
But the code did pass the planned additional tests (queue browsing and
message selection, creation of connection factory from LDAP-administered
definitions), plus many unit tests.
It proved to be usable in basic and some more advanced scenarios.
Therefore I would consider it to be in beta / advanced beta state.


I'll let you get back to me regarding whether a code grant is needed, and
whether it would be safe to keep the excerpts from IBM documentation.

I'll also let you tell me how I should "officially" provide the source
package. Should I create a JIRA issue and attach the code as suggested in
the "Submitting patches" section of the "Contributing" page ?

Best regards,

Stéphane

On Wed, Dec 16, 2015 at 5:08 AM, Jim Gomes <jg...@apache.org> wrote:

> Hi Stephane,
>
> My apologies for the delay in replying. It's a busy time of year for me.
>
> I think basing it on the EMS code was a good idea. I'm not sure where the
> line is when a regular license assignment is sufficient, or when a code
> grant is required.  It's a judgment call, so I think I'll ask around as to
> what the best way to proceed would be.  When I originally wrote the EMS
> provider implementation, it was enough new code that they wanted a code
> grant.  Since you based your work on that, would you estimate how many
> lines of new code there is?  Since you developed this on your own time, and
> were careful to keep it separate, that should make things much simpler.
>
> Regarding the IBM documentation reference, I'm not sure if it would need
> to be reworked, or whether it would fall under the Fair Use copyright
> exclusion.  I think the Apache legal department could help out with that.
>
> I think it's OK to have the code for the 3 types of messaging servers
> included, even if you can't test them all.  There might be others who can
> take what you did and test it out and submit reports back.  The code
> doesn't have to be 100% functional.  This is a great start.
>
> You suggested timeline for features and work sounds great.  I think we can
> get the base functionality imported and then moving forward as you have
> time.
>
> Thanks again for your willingness to contribute!
>
> Best,
> Jim
>
>
>
> On Wed, Dec 9, 2015 at 12:54 PM, Stéphane Ramet <
> stephane.ramet.sr@gmail.com> wrote:
>
>> Hi Jim,
>>
>> Thank you for your reply.
>>
>> Mainly, this is the amount of code I developed, based on and adapted from
>> the EMS provider code :
>>
>> .:
>> -rwxrwxrwa    4102 Nov 29 02:02 BytesMessage.cs
>> -rwxrwxrwa    1346 Dec  5 15:59 CommonAssemblyInfo.cs
>> -rwxrwxrwa   11859 Nov 30 18:16 Connection.cs
>> -rwxrwxrwa   51947 Nov 30 23:03 ConnectionFactory.cs
>> -rwxrwxrwa    2828 Nov 28 18:08 ConnectionMetaData.cs
>> -rwxrwxrwa   17262 Nov 29 23:32 Destination.cs
>> -rwxrwxrwa    6262 Nov 30 13:41 InitialContext.cs
>> -rwxrwxrwa   15370 Nov 30 22:51 MapMessage.cs
>> -rwxrwxrwa   17732 Nov 29 22:39 Message.cs
>> -rwxrwxrwa    5029 Nov 30 14:05 MessageConsumer.cs
>> -rwxrwxrwa    9775 Nov 30 18:00 MessageProducer.cs
>> -rwxrwxrwa    9196 Nov 30 14:11 MessageProperties.cs
>> -rwxrwxrwa    1927 Nov 29 22:12 ObjectMessage.cs
>> -rwxrwxrwa    2068 Nov 29 02:01 Queue.cs
>> -rwxrwxrwa    2696 Nov 28 18:57 QueueBrowser.cs
>> -rwxrwxrwa   12681 Nov 30 14:36 Session.cs
>> -rwxrwxrwa   11210 Nov 29 02:02 StreamMessage.cs
>> -rwxrwxrwa    1856 Nov 27 00:10 TemporaryQueue.cs
>> -rwxrwxrwa    1856 Nov 27 00:10 TemporaryTopic.cs
>> -rwxrwxrwa    2156 Nov 29 02:02 TextMessage.cs
>> -rwxrwxrwa    2006 Nov 29 02:01 Topic.cs
>> drwxrwxrwx       0 Nov 29 01:48 Util
>>
>> ./Util:
>> -rwxrwxrwa    4919 Nov 26 15:42 Dispatcher.cs
>> -rwxrwxrwa    4676 Nov 29 21:48 ExceptionUtil.cs
>> -rwxrwxrwa   13622 Nov 29 01:48 IntrospectionSupport.cs
>> -rwxrwxrwa    1792 Nov 18 21:22 UriAttributeAttribute.cs
>> -rwxrwxrwa   40560 Nov 30 17:52 XMSConvert.cs
>> -rwxrwxrwa   36672 Nov 29 22:21 XMSEnum.cs
>>
>> I'm not exactly sure to understand what a Code Grant means. I suppose it
>> means that, once accepted, I won't be able to claim any intellectual
>> property on the code and therefore won't be able to prevent anyone from
>> freely using and modifying it ? If so, it won't be a problem.
>>
>> On the legal aspects, since all this was produced on my spare time and
>> not on company time, on my hardware and not on my company's or customer's
>> hardware, I believe my company or customer can't claim any intellectual
>> property on the code. Only testing of the compiled DLL's requires me to use
>> some of my customer's resources, as if they had been downloading the DLL
>> from any web site. I don't think it gives them any intellectual rights on
>> the code either. Furthermore, I don't think any clause in my employment
>> contract prevents me from contributing to open source communities.
>> Therefore I think there shouldn't be any legal complications on my side.
>> I'll double-check anyway.
>>
>> Still on the legal aspects : in order to make things fast and easy, I
>> referenced and copy-pasted parts of IBM's documentation in the code.
>>
>> eg.
>> //
>> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.msc.doc/prp_jms_ibm_encoding.htm?lang=en
>> /// <summary>
>> /// How numerical data in the body of the message is represented when the
>> XMS
>> /// client forwards the message to its intended destination.
>> /// </summary>
>> public enum Encoding
>> {
>>     /// <summary>
>>     /// Normal integer encoding.
>>     /// </summary>
>>     IntegerNormal = MQC.MQENC_INTEGER_NORMAL,
>> ...
>>
>> This possibly would need to be reworked. Would it ?
>>
>> You certainly know that the IBM XMS client can address more than just
>> WebSphere MQ queue managers ("WMQ"), but also WebSphere Application
>> Server Service Integration Buses ("WPM") or IBM Integration Buses
>> through WebSphere MQ Real-Time Transport ("RTT"). I implemented all
>> properties for supporting the 3 types of messaging servers. But in my
>> environment, I will only be able to test the "WMQ" part. How should the
>> other two be handled (tagged ? commented out ? ...) until me or someone
>> else gets a chance to test them ?
>>
>> XMS also supports creation of ConnectionFactory and other objects from
>> repositories of administered objects. I implemented the logic to do so, but
>> similarly I'm not sure how much of this I will be able to test.
>>
>> Last point : what timelines would you expect ? My customer's expectations
>> are to be able to handle basic use cases as described in my previous mail.
>> This has already been tested OK. My personal aim is to cover more WMQ
>> functionalities by the end of the year (at least queue browsing,
>> selectors and topics), possibly including some NUnit tests (I didn't write
>> any yet). Is that OK with you ?
>>
>> Best regards,
>>
>> Stephane Ramet
>>
>> On Wed, Dec 9, 2015 at 1:38 AM, Jim Gomes <jg...@apache.org> wrote:
>>
>>> Hi Stephane,
>>>
>>> Thank you for the e-mail. The Apache.NMS.MSMQ provider is still
>>> officially supported, but hasn't had any active development recently. It
>>> should still compile with the latest Apache.NMS 1.7.1 (or 1.7.2) API, using
>>> the trunk code base. It hasn't been tagged or branched in a while, so
>>> you'll need to build your own.
>>>
>>> For the Apache.NMS.EMS provider, it has been branched into 1.7.x, and it
>>> will compile against the 1.7.1 or 1.7.2 API. Tagged versions have been
>>> created for this, but they haven't been officially released yet.
>>>
>>> Having an XMS provider would be great! That's a work item that I entered
>>> into JIRA a while ago as a potential project to work on. If you would like
>>> to contribute it back to the project, I can help with that process. If it
>>> is a significant chunk of code, we will probably need to go through the
>>> process of a Code Grant. This is a little more involved and would require
>>> the legal department to be involved. It's not difficult, but the Apache
>>> Foundation wants to make sure that intellectual property rights are clearly
>>> preserved.  How much new code is involved with the XMS provider you've been
>>> working on?  I hope you aren't put off by the possibility of a Code Grant.
>>> When I first started working with NMS, I went through the process, and it
>>> was fairly straightforward.
>>>
>>> I look forward to working with you on this.
>>>
>>> Best Regards,
>>> Jim
>>>
>>>
>>> On Tue, Dec 8, 2015 at 12:20 PM, Stéphane Ramet <
>>> stephane.ramet.sr@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> for one of my customers', I need to provide messaging capabilities to
>>>> some .NET applications in an environment where integration platforms are
>>>> currently moved from WebsphereMQ and MSMQ to Tibco.
>>>>
>>>> I was considering Apache.NMS for decoupling the application code from
>>>> the underlying transport, but browsing through the site I found quite a
>>>> number of broken links (MSMQ and EMS provider downloads are compiled in
>>>> version 1.1, which is not available online any more ; Sandcastle Styles are
>>>> not available any more either).
>>>>
>>>> Therefore I was wondering what the status for this project is, or at
>>>> least for those providers. Is it still active ?
>>>>
>>>> I downloaded the source code from SVN and am currently testing it. The
>>>> MSMQ provider works fine in my first basic use cases (simple message
>>>> production, transacted production, synchronous and asynchronous
>>>> consumption). I should get access to a sand box environment this week to
>>>> test the EMS provider.
>>>>
>>>> I also developed an XMS provider and started testing it (same basic use
>>>> cases with the same test code as for MSMQ).
>>>>
>>>> I can possibly contribute a .NET 4 source code, reasonably tested with
>>>> XMS 8.0.0. Please tell me if it is of any interest, and guide me on how to
>>>> contribute on that issue, since I'm quite new to that community, and to
>>>> open source in general.
>>>>
>>>> Regards,
>>>>
>>>> Stephane Ramet
>>>>
>>>
>>>
>>
>