[LDAP API] encoding refactoring status, 2

[Emmanuel Lécharny <el...@gmail.com>]

Hi !

I'm done with controls, and I'm currently adding tests for messages that 
have controls to check they all are properly handled. In the process, I 
have found *many* errors (like some controls that are simply nit working 
at all, not being able to be decoded...)

The next step will be the extended operations, then the intermediate 
operation (we have only one).

Once done, I'll be able to phase out the existing encoder to use the 
reverse one.

I hope to be done with this by the end of this week. When done, the next 
big step will be the removal of decorator.

More to come....

Re: [LDAP API] encoding refactoring status, 7

[Emmanuel Lécharny <el...@gmail.com>]

Hi !

Almost there... ALl the server-iunteg tests are passing but 4 
replication tests.


I had to fight against two nasty bugs yesterday, and it took me a while 
to understand what was going bad.

The fist big bug was difficult to ffix, because it was difficult to 
isolate. The tests were working in the IDE but everything was blocking 
when ran using 'mvn clean install', so it took a while to point out 
which exact test was blocking. At the end of the day, it was a bad PDU 
that generated an exception (on purpose), with a NoticeOfDisconnect not 
being handled the right way server side: the connection was shutdown too 
early on the server, and the NoD was never sent to the client, which was 
hanging forever.

The second bug was almost 10 years old : when an operation was aborted 
server side for any reason - in that case, we configured the server to 
refuse a request that was bigger than a given size -, a NoD was sent and 
never handled by the client, which simply timed out. That was kind of 
passing because the time out was low (30 seconds), and get unnoticed. 
With the changes I applied in the previous bug, the NoD generated a NPE, 
and then the bug was apparent.

So now, NoD are properly handled on both side (client and server).

Some of the other issues I faced with this refactoring is the fact that 
some of the extended response simply don't have a responseName, which 
makes it complicated to find their factory: I finally opted to stored 
the extendedRequest in the ExtendedFuture data structure, so when we 
receive the un-typed response, I just had to check the request to know 
which kind of response it was.

All of these complications are related to the fact that the API must not 
only support extended ops and controls we know of, but also those we 
have no clue about: we should be able to send an unknown control or 
extended operation, assuming the value is properly encoded and decoded 
by the user.


So when those 4 last failing tests will be fixed, I think we will be 
good to go - most of the checks are done in server-integ -. Not sure we 
will be ok this year, and we will definitively not have a release in 
2018, but that may be ok for the very beginning of 2019 ;-)


Happy new year !

[LDAP API] encoding refactoring status, 6

[Emmanuel Lécharny <el...@gmail.com>]

Hi!


As usual, it always takes way longer than expected to get things done :/


I'm facing a road block atm, in one test: search with transactions 
(where we add 10000 entries using transactions). The problem is that RFC 
5805 specifies that externded responses don't have a responseName (the 
ExtnededResponse OID), which makes it quite tricky to retrieve the 
associated factory (as it's stored in a Map<OID, Factory>).
I implemented a workaround : I temporarily inject the responseName to 
get the extended response properly encoded, then remove it, and on the 
API side, the extendedFuture now contains the extendedRequest, and I 
match the request MessageID with the extendedResponse MessageId, to 
retrieve the responseName, then I can decode the response.

It's quite convoluted...

I'm hitting the road today, so I won't make a lot of progress.

[LDAP API] encoding refactoring status, 5

[Emmanuel Lécharny <el...@gmail.com>]

Hi !

post-X-mas update...

All the LDAP API tests are passing green after a complete decorator 
removal (except from the DSML code), but a few OSGi ones. Which means 
the basic code is working. I will fix the OSGi tests today  (2 failures, 
5 errors), then check the server.

I'm confident it's going to be done very soon... We are talking about a 
49 000 lines of git diff here, so it's not a small refactoring. From the 
API user PoV, though, it should be pretty transparent.

I had to differ the JNDI code removal, it's heavily used in the server 
tests, and changing that would have taken way to much time. Also it can 
be done in a next iteration.

What will remain to be done as soon as I'm done with both the server and 
the API will be to merge those changes with Stefan's changes. It should 
go smoothly though.


More coming soon !

Re: [LDAP API] encoding refactoring status, 4

[Emmanuel Lécharny <el...@gmail.com>]

On 23/12/2018 18:16, Stefan Seelmann wrote:
> On 12/23/18 4:34 PM, Emmanuel Lécharny wrote:
>> Touching base, soon. I'm done with all the standard operations, all the
>> controls (the 23 of them...), i'm almost done with the Intermediate
>> Response (SyncIInfoValue) and then I will have to process the
>> ExtendedOperations. This is easier and easier.
>>
>> I'm talkingh about Decorator romovel here (the reverse encoding is fully
>> done). If I'm not to buys with external tasks (like in familly tasks
>> ;-), I may be done tonite with extended operations.
>>
>> Once completed, I have a couple of cleanup tasks to do - not really big
>> -, then there are a few things I'd like to do before cutting a release :
>>
>> - make the Asn1Decoder completely stateless. Currently, we store a bit
>> of status in it, which deserves to be moved to the Container. This
>> should not take too long.
>>
>> - then I want to make the Asn1Buffer allocated in a Thread Local
>> Storage, to avoid having to allocate it for every operation that
>> requires some encoding.
>>
>> - I want to remove all the JNDI part of the API (but that may be
>> post-poned to the next release)
>>
>> - last, not least, I think the Mutable/Immutable SchemaObject is a mess.
>> I'd like to rethink this part a bit. Same thing, might be for a next
>> release.
>>
>>
>> If you have anything you'd want to see in the coming release, please let
>> me know !
> You mean a milestone release, right?

Correct.


> Then all fine from my side, current
> API and server master/snapshot work well with Studio.


I don't think the changes I'm currently doing are going to impact Studio.

>
> For a final 2.0 API release I'd like to suggest changes in SSL/TLS
> certificate handling, I'll write about that in a separate thread.

Sure. The SSL/TLS certificate support in the API is a bit defficient...


Waiting to read you :-)

Re: [LDAP API] encoding refactoring status, 4

[Stefan Seelmann <ma...@stefan-seelmann.de>]

On 12/23/18 4:34 PM, Emmanuel Lécharny wrote:
> Touching base, soon. I'm done with all the standard operations, all the
> controls (the 23 of them...), i'm almost done with the Intermediate
> Response (SyncIInfoValue) and then I will have to process the
> ExtendedOperations. This is easier and easier.
> 
> I'm talkingh about Decorator romovel here (the reverse encoding is fully
> done). If I'm not to buys with external tasks (like in familly tasks
> ;-), I may be done tonite with extended operations.
> 
> Once completed, I have a couple of cleanup tasks to do - not really big
> -, then there are a few things I'd like to do before cutting a release :
> 
> - make the Asn1Decoder completely stateless. Currently, we store a bit
> of status in it, which deserves to be moved to the Container. This
> should not take too long.
> 
> - then I want to make the Asn1Buffer allocated in a Thread Local
> Storage, to avoid having to allocate it for every operation that
> requires some encoding.
> 
> - I want to remove all the JNDI part of the API (but that may be
> post-poned to the next release)
> 
> - last, not least, I think the Mutable/Immutable SchemaObject is a mess.
> I'd like to rethink this part a bit. Same thing, might be for a next
> release.
> 
> 
> If you have anything you'd want to see in the coming release, please let
> me know !

You mean a milestone release, right? Then all fine from my side, current
API and server master/snapshot work well with Studio.

For a final 2.0 API release I'd like to suggest changes in SSL/TLS
certificate handling, I'll write about that in a separate thread.

Thanks Emmanuel!

Kind Regards,
Stefan

Re: [LDAP API] encoding refactoring status, 4

[Emmanuel Lécharny <el...@gmail.com>]

Hi !

Touching base, soon. I'm done with all the standard operations, all the 
controls (the 23 of them...), i'm almost done with the Intermediate 
Response (SyncIInfoValue) and then I will have to process the 
ExtendedOperations. This is easier and easier.

I'm talkingh about Decorator romovel here (the reverse encoding is fully 
done). If I'm not to buys with external tasks (like in familly tasks 
;-), I may be done tonite with extended operations.

Once completed, I have a couple of cleanup tasks to do - not really big 
-, then there are a few things I'd like to do before cutting a release :

- make the Asn1Decoder completely stateless. Currently, we store a bit 
of status in it, which deserves to be moved to the Container. This 
should not take too long.

- then I want to make the Asn1Buffer allocated in a Thread Local 
Storage, to avoid having to allocate it for every operation that 
requires some encoding.

- I want to remove all the JNDI part of the API (but that may be 
post-poned to the next release)

- last, not least, I think the Mutable/Immutable SchemaObject is a mess. 
I'd like to rethink this part a bit. Same thing, might be for a next 
release.


If you have anything you'd want to see in the coming release, please let 
me know !


Thanks!

[LDAP API] encoding refactoring status, 3

[Emmanuel Lécharny <el...@gmail.com>]

Hi !

I'm done with the reverse encoding refactoring ! It took some time, 
fixing the replication part was a bit tough. I also have removed the use 
of Decorator when writing a message (a bad move I made 4 months ago).

There are two next steps:

- first, use a thread-local-storage to keep the buffer used to encode 
messages, in order to avoid allocating it over and over (that was the 
root reason I worked on a reverse-encoding). I have done some 
preliminary investigation, and had it working, but I have to get it done 
properly

- then remove the decorators (again... But remember I have lost my 
temporary commit last month :/)


Once done, we should be able to cut a 2.0 and resume normal operations, 
with many fixes to be applied on the server itself. I hope to be able to 
deliver this 2.0 release as a Xmas present :-)