[DISCUSS] Thoughts on Jacob refactoring

[Hadrian Zbarcea <hz...@gmail.com>]

Tammo was kind enough to spend some time with me at ApacheCon EU and 
educate me on a few things in ODE (which happend in the context of a few 
other discussions related to possible future enhancements in Camel; 
thanks Guillaume for your ideas and pragmatic approach). So I started a 
bit of research and I see very interesting opportunities for using not 
as much ODE, but Jacob in Camel. I'll take this on the Camel lists soon, 
for now I tried to understand Jacob and made a few improvements along 
the way (on a github fork [1]).

I'd like to share comments and proposals and get a feel of what would 
sound as decent, feasible to the community (or well, nonsense, but 
hopefully not).

1. Jacob is a framework, ODE (rest of it) is an application. As they 
potentially have different uses and life cycles, it would make sense to 
me to separate their release cycles.
2. Jacob dependencies are a bit messed up [2], jacob doesn't really need 
to depend on utils. The latter dependends on the world, which may not be 
necessary. Removing jacob dependencies on utils is already fixed in my fork.
3. There is not enough unit testing imho. There are also a few failing 
tests, not many. In my fork I explicitly excluded failing tests from 
executing, to ensure that my refactoring does not introduce any 
regression issues. Obviously not a guarantee, but that's all I had to 
work with.
4. The apt based code generation should be improved [3]. On my fork I 
already fixed that and upgraded to a jsr-269 based code generation. 
However, looking more at the code it looks like the code generation is 
not even necessary. First of all the generated Channel interfaces are 
never instantiated. What gets instantiated are CommChannel<T extends 
Channel> and then dynamic Proxy are used (in a quite clever way I should 
add). I guess the generation was considered necessary because with Java 
generics you cannot extend a generic type, because of erasure, unlike 
other languges (C++ templates/metaprograming for instance). However, 
unique types are not necessary (function served by the generated Channel 
interfaces), dynamic Proxies can handle multiple interfaces very well, 
without an existing type (for the curious, I have a unit test 
demonstrating that). So I fixed that on my fork, but ran into a bunch of 
side effects related to the fact that the api expects to use generated 
types that extend Channel. A better
solution however is for the channel interface to extend a ChannelType 
(empty) interface, as opposed to being annotated with a @ChannelType and 
use this ChannelType as the base interface in the api. That requires 
small api changes with ripple effect across the whole code base which 
will take me a few iterations to get right (I prefer to do it in small 
increments vs one big bang). Then I'll need to change to a solution 
based on delegation vs inheritance and make the generated 
ChannelListeners unnecessary. After that the whole jsr-269 based 
generation will become obsolete.
5. Logging is kinda ugly with all those guards. I think there are better 
solutions today to avoid the evaluation of expression passed to the 
logging methods when the log level does not allow for actual logging. 
I'd suggest moving to slf4j as many other project did. I am planning on 
doing that if no objections. Related to this, I am not sure if i18n is 
necessary for all the components, jacob in particular. It's such a low 
level framework that i18n support for logging doesn't seem necessary.
6. The way stats are collected and (not) used could definitely be improved.

... next steps
7. Tammo was talking about improvements around the omodel. We talked 
about using maybe some digests (sha1) for references instead of running 
counters to maintain consistency.
8. Tammo also mentioned some ideas related to improving correlation (I 
still need to fully understand that)
9. Related to Camel. I will be looking for an abstraction for the Jacob 
engine. In particular the concept of Continuation is missing in Camel 
and badly needed. For those familiar with Camel, the unit of work is 
supposed to be similar, but not really. In Camel, processors call onto 
the next ones leading to (sometimes ridiculously) deep stacks. An 
abstraction that would suit both models would help with a more smooth 
integration of Jacob with Camel. I don't know yet for instance if a 
different way of generating routes from the DSL would help, or even be 
required. Anyway, something for the Camel lists.

I bet there will be more improvements and ideas in the near future but I 
wanted to share this for now and hopefully get some feedback (or a slap 
on the wrist if something doesn't sound quite right).

Cheers,
Hadrian


[1] https://github.com/hzbarcea/ode
[2] https://issues.apache.org/jira/browse/ODE-978
[3] https://issues.apache.org/jira/browse/ODE-979


-- 
Hadrian Zbarcea
Principal Software Architect
Talend, Inc
http://coders.talend.com/
http://camelbot.blogspot.com/

Re: [DISCUSS] Thoughts on Jacob refactoring

[Paul Brown <pa...@gmail.com>]

Hi, Hadrian --

No slaps on the wrist from me!  You're correctly observing that the Jacob
code is probably on the order of 8 years old in terms of when it was first
developed, and while the libraries around the code have changed a lot over
that period, the way that the code uses them probably has not.    Moreover,
I'm certain that in the more modern world there are probably any number of
viable alternative techniques to achieve some of the same goals.

For a random bit of history, the ideas in Jacob came from general thoughts
about continuations in the context of creating resumable "flows" in Java,
and the application to a implementing BPEL engine was opportunistic.  If
Maciej's still lurking around on the list, he might have some notes on the
original set of motivating examples and other frameworks we looked at.
 (Brakes (http://people.cs.kuleuven.be/~eddy.truyen/BRAKES/brakes.html) and
TyCO (http://www.dcc.fc.up.pt/~tyco/) come to mind.)

I'll follow along as you work on your fork on Github.

-- Paul


On Fri, Dec 7, 2012 at 2:05 PM, Hadrian Zbarcea <hz...@gmail.com> wrote:

> Tammo was kind enough to spend some time with me at ApacheCon EU and
> educate me on a few things in ODE (which happend in the context of a few
> other discussions related to possible future enhancements in Camel; thanks
> Guillaume for your ideas and pragmatic approach). So I started a bit of
> research and I see very interesting opportunities for using not as much
> ODE, but Jacob in Camel. I'll take this on the Camel lists soon, for now I
> tried to understand Jacob and made a few improvements along the way (on a
> github fork [1]).
>
> I'd like to share comments and proposals and get a feel of what would
> sound as decent, feasible to the community (or well, nonsense, but
> hopefully not).
>
> 1. Jacob is a framework, ODE (rest of it) is an application. As they
> potentially have different uses and life cycles, it would make sense to me
> to separate their release cycles.
> 2. Jacob dependencies are a bit messed up [2], jacob doesn't really need
> to depend on utils. The latter dependends on the world, which may not be
> necessary. Removing jacob dependencies on utils is already fixed in my fork.
> 3. There is not enough unit testing imho. There are also a few failing
> tests, not many. In my fork I explicitly excluded failing tests from
> executing, to ensure that my refactoring does not introduce any regression
> issues. Obviously not a guarantee, but that's all I had to work with.
> 4. The apt based code generation should be improved [3]. On my fork I
> already fixed that and upgraded to a jsr-269 based code generation.
> However, looking more at the code it looks like the code generation is not
> even necessary. First of all the generated Channel interfaces are never
> instantiated. What gets instantiated are CommChannel<T extends Channel> and
> then dynamic Proxy are used (in a quite clever way I should add). I guess
> the generation was considered necessary because with Java generics you
> cannot extend a generic type, because of erasure, unlike other languges
> (C++ templates/metaprograming for instance). However, unique types are not
> necessary (function served by the generated Channel interfaces), dynamic
> Proxies can handle multiple interfaces very well, without an existing type
> (for the curious, I have a unit test demonstrating that). So I fixed that
> on my fork, but ran into a bunch of side effects related to the fact that
> the api expects to use generated types that extend Channel. A better
> solution however is for the channel interface to extend a ChannelType
> (empty) interface, as opposed to being annotated with a @ChannelType and
> use this ChannelType as the base interface in the api. That requires small
> api changes with ripple effect across the whole code base which will take
> me a few iterations to get right (I prefer to do it in small increments vs
> one big bang). Then I'll need to change to a solution based on delegation
> vs inheritance and make the generated ChannelListeners unnecessary. After
> that the whole jsr-269 based generation will become obsolete.
> 5. Logging is kinda ugly with all those guards. I think there are better
> solutions today to avoid the evaluation of expression passed to the logging
> methods when the log level does not allow for actual logging. I'd suggest
> moving to slf4j as many other project did. I am planning on doing that if
> no objections. Related to this, I am not sure if i18n is necessary for all
> the components, jacob in particular. It's such a low level framework that
> i18n support for logging doesn't seem necessary.
> 6. The way stats are collected and (not) used could definitely be improved.
>
> ... next steps
> 7. Tammo was talking about improvements around the omodel. We talked about
> using maybe some digests (sha1) for references instead of running counters
> to maintain consistency.
> 8. Tammo also mentioned some ideas related to improving correlation (I
> still need to fully understand that)
> 9. Related to Camel. I will be looking for an abstraction for the Jacob
> engine. In particular the concept of Continuation is missing in Camel and
> badly needed. For those familiar with Camel, the unit of work is supposed
> to be similar, but not really. In Camel, processors call onto the next ones
> leading to (sometimes ridiculously) deep stacks. An abstraction that would
> suit both models would help with a more smooth integration of Jacob with
> Camel. I don't know yet for instance if a different way of generating
> routes from the DSL would help, or even be required. Anyway, something for
> the Camel lists.
>
> I bet there will be more improvements and ideas in the near future but I
> wanted to share this for now and hopefully get some feedback (or a slap on
> the wrist if something doesn't sound quite right).
>
> Cheers,
> Hadrian
>
>
> [1] https://github.com/hzbarcea/**ode <https://github.com/hzbarcea/ode>
> [2] https://issues.apache.org/**jira/browse/ODE-978<https://issues.apache.org/jira/browse/ODE-978>
> [3] https://issues.apache.org/**jira/browse/ODE-979<https://issues.apache.org/jira/browse/ODE-979>
>
>
> --
> Hadrian Zbarcea
> Principal Software Architect
> Talend, Inc
> http://coders.talend.com/
> http://camelbot.blogspot.com/
>