You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Tomohisa Igarashi <tm...@gmail.com> on 2016/11/07 03:06:35 UTC
Re: Adding type awareness in Camel route
Hi,
I filed a JIRA for this one:
https://issues.apache.org/jira/browse/CAMEL-10447
and added a camel-example-transformer with some bug fixes:
https://github.com/igarashitm/camel/tree/CAMEL-10447
https://github.com/igarashitm/camel/commit/1c9bb6d8f23e0d96e942161f8ffc821845d99c59#diff-02f2b5457a9643a58d973ef7e971d405
https://github.com/igarashitm/camel/commit/1c9bb6d8f23e0d96e942161f8ffc821845d99c59#diff-b64bb94ed2382215eef0598fc79aa9bf
I'll submit a PR once I finish the document bits. I haven't integrated with existing data type awareness on REST DSL yet, but I believe that won't be too complex. Anyway that is a next thing to do.
Thanks,
Tomo
On 10/26/2016 10:52 PM, Tomohisa Igarashi wrote:
> Hi Claus,
>
> Thanks for the feedback!
>
> On 10/26/2016 09:39 PM, Claus Ibsen wrote:
>> On Mon, Oct 17, 2016 at 4:35 AM, Tomohisa Igarashi
>> <tm...@gmail.com> wrote:
>>> Hi,
>>>
>>> I'd like to resume the discussion about this. I still think 3.0 would be the
>>> best target to get this feature fully supported, but to achieve it in better
>>> shape, I'd like to have it in 2.19 as well as an experimental, ask feedback
>>> and then reflect those for 3.0 full support. Fortunately this is purely an
>>> addition to existing features, i.e. is not breaking any existing API. What
>>> do you think?
>>>
>>
>> Having community feedback would be great.
>>
>> I wonder if you would mind creating a little example in the examples
>> that uses this new feature. Then its easier for users to look at and
>> better understand what its about.
>
> OK sure, I'll add an example for this feature.
>
>> Also when we do DSL changes/additions we tend to write unit tests in
>> camel-core, and then reuse those tests in camel-spring where we extend
>> the unit test from camel-core-test and then provide the routes in XML
>> DSL instead of java code. Then the unit test is the same and ensures
>> the DSL works in both worlds (Java and XML). I am not sure if you have
>> this in your commits?
>>
>> You can take a look at the existing tests in camel-spring to find some
>> examples of this practice.
>
> Yep, actually SpringTransformerTest inherits test methods from TransformerTest with that practice :)
> I'll add more unit tests in addition to that.
>
>> Down the road we do have some kind of typeawareness with the rest-dsl
>> where you can also specify in and output types which the swagger
>> api-doc uses. And it does some automatic binding to/from
>> pojo/xml/json. So we may have some overlap here, and may ponder if
>> there is something we can eventually make use the new typeware stuff
>> or meet in the middle or something. However maybe its better to take a
>> look at that after the other stuff is more ready.
>
> That sounds like a thing I need to look into, at least need to understand what is available right now. Hopefully there's a good meeting point with it.
>
>> Also in Java DSL you may want to allow to specify a java type using
>> the Class as type instead of having to type the FQN in a String. But
>> mind in XML DSL this is not possible where you specify a String
>> attribute for the type.
>
> Good idea, I'll add that too!
>
>>> Thanks,
>>> Tomo
>>>
>>>
>>> On 09/17/2016 10:20 PM, Tomohisa Igarashi wrote:
>>>>
>>>> Hi Claus,
>>>>
>>>> Thanks for the reply. Sure that's fine, I agree 3.0 would be better to be
>>>> targeted than 2.x as this introduces some schema updates.
>>>>
>>>> Including this one, I'm always looking for the chance to make any
>>>> contribution to Camel. If there's anything I can help please let me know.
>>>>
>>>> Thanks,
>>>> Tomo
>>>>
>>>> On 09/17/2016 06:33 PM, Claus Ibsen wrote:
>>>>>
>>>>> Hey
>>>>>
>>>>> Can we take this discussion post Camel 2.18 release.
>>>>>
>>>>> We are working on the last details to get it ready, and its our main
>>>>> focus to get this new release out.
>>>>>
>>>>> After this release we will pickup talks about the next releases
>>>>> whether that is 2.19 or 3.0, and for the latter what the broad goals
>>>>> of that is. What you talk about seems more of a 3.0 candidate to me,
>>>>> than on 2.x.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 16, 2016 at 6:50 AM, Tomohisa Igarashi
>>>>> <tm...@gmail.com> wrote:
>>>>>>
>>>>>> Hi Camel developers,
>>>>>>
>>>>>> I'd like to propose an enhancement on handling data types of Camel
>>>>>> message
>>>>>> contents. To start a smooth discussion I implemented the idea first:
>>>>>> https://github.com/igarashitm/camel/tree/contract-based-type-awareness
>>>>>>
>>>>>> And these testcases demonstrates the declarative transformer usage
>>>>>> according
>>>>>> to the declared data types:
>>>>>> [Java DSL]
>>>>>>
>>>>>> https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-c14a7e8e88a6e41492946e1537bfb1cf
>>>>>> [Spring DSL]
>>>>>>
>>>>>> https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-b2506c84ddde91438fc6374039e21534
>>>>>>
>>>>>> This adds input/output content type declaration on from and all other
>>>>>> processors. It also introduces well-known Exchange properties,
>>>>>> INPUT_TYPE
>>>>>> and OUTPUT_TYPE which are used to specify the current message content
>>>>>> type.
>>>>>> The data type is URN like string starts with scheme, like
>>>>>> java:org.example.ItemA or xml:{org.example.xml}ItemA.
>>>>>>
>>>>>> If the content type is declared via inputType/outputType/contract, the
>>>>>> ContractProcessor wraps the actual processor and process
>>>>>> transformation/validation according to the type, say if INPUT_TYPE
>>>>>> exchange
>>>>>> property has xml:{org.example.xml}ItemA and the declared inputType is
>>>>>> xml:{org.example.xml}ItemB, then it transforms
>>>>>> xml:{org.example.xml}ItemA
>>>>>> content into xml:{org.example.xml}ItemB. The <transformers> element
>>>>>> which is
>>>>>> introduced right under the <camelContext> is the one to declare the
>>>>>> mappings
>>>>>> between transformer implementation and those from/to data type. I
>>>>>> implemented only transformer first, but validator would be brought in in
>>>>>> a
>>>>>> same way. This way allows users to make data types visible in the route
>>>>>> definition and keep the transformation/validation apart from route
>>>>>> definition itself.
>>>>>>
>>>>>> The most important thing is that the ContractProcessor is involved only
>>>>>> when
>>>>>> content type is explicitly declared in a route definition, so that it
>>>>>> never
>>>>>> breaks existing camel routes. Ofcourse programatic
>>>>>> transformations/validations we're doing today are still fully available.
>>>>>> It's purely an addition to the existing camel features.
>>>>>>
>>>>>> Any thoughts? Does it sound acceptable to get merged into camel? ANY
>>>>>> feedback would be really appreciated!
>>>>>>
>>>>>> Thanks,
>>>>>> Tomo
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>
>>
>>
Re: Adding type awareness in Camel route
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
So the PR from Tomo was a bit overwhealming.
I have talked with Tomo to do this in smaller steps so we can all
follow where this is taking Camel.
At first step is something like
- type awareness to route level (similar to rest-dsl binding)
- example in xml
- example in java
- this feature is totally opt-on
- allow to define input and output as just "xml" "json" or "java" etc
to just indicate its one of these kind, but maybe without reference to
a xml schema / namespace etc / ditto for json. But being able to just
tell the Camel developer that this is a route that receives XML and
returns JSon is a good win. This allows developers to easier maintain
this code in the future to be able to see this.
And then step 2 can be
- can we reuse/align this with rest-dsl binding
- feedback from community, rinse and repeat
- jmx and some other apis to expose this for tooling / management
And for following steps
- should we have type awareness on processor level also? And if so we
need some non invasive general hook to perform this, so we dont have
to scatter code around. However this is a bridge to cross in the
future.
On Mon, Nov 7, 2016 at 4:06 AM, Tomohisa Igarashi <tm...@gmail.com> wrote:
> Hi,
>
> I filed a JIRA for this one:
> https://issues.apache.org/jira/browse/CAMEL-10447
>
> and added a camel-example-transformer with some bug fixes:
> https://github.com/igarashitm/camel/tree/CAMEL-10447
> https://github.com/igarashitm/camel/commit/1c9bb6d8f23e0d96e942161f8ffc821845d99c59#diff-02f2b5457a9643a58d973ef7e971d405
> https://github.com/igarashitm/camel/commit/1c9bb6d8f23e0d96e942161f8ffc821845d99c59#diff-b64bb94ed2382215eef0598fc79aa9bf
>
> I'll submit a PR once I finish the document bits. I haven't integrated with
> existing data type awareness on REST DSL yet, but I believe that won't be
> too complex. Anyway that is a next thing to do.
>
> Thanks,
> Tomo
>
>
> On 10/26/2016 10:52 PM, Tomohisa Igarashi wrote:
>>
>> Hi Claus,
>>
>> Thanks for the feedback!
>>
>> On 10/26/2016 09:39 PM, Claus Ibsen wrote:
>>>
>>> On Mon, Oct 17, 2016 at 4:35 AM, Tomohisa Igarashi
>>> <tm...@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'd like to resume the discussion about this. I still think 3.0 would be
>>>> the
>>>> best target to get this feature fully supported, but to achieve it in
>>>> better
>>>> shape, I'd like to have it in 2.19 as well as an experimental, ask
>>>> feedback
>>>> and then reflect those for 3.0 full support. Fortunately this is purely
>>>> an
>>>> addition to existing features, i.e. is not breaking any existing API.
>>>> What
>>>> do you think?
>>>>
>>>
>>> Having community feedback would be great.
>>>
>>> I wonder if you would mind creating a little example in the examples
>>> that uses this new feature. Then its easier for users to look at and
>>> better understand what its about.
>>
>>
>> OK sure, I'll add an example for this feature.
>>
>>> Also when we do DSL changes/additions we tend to write unit tests in
>>> camel-core, and then reuse those tests in camel-spring where we extend
>>> the unit test from camel-core-test and then provide the routes in XML
>>> DSL instead of java code. Then the unit test is the same and ensures
>>> the DSL works in both worlds (Java and XML). I am not sure if you have
>>> this in your commits?
>>>
>>> You can take a look at the existing tests in camel-spring to find some
>>> examples of this practice.
>>
>>
>> Yep, actually SpringTransformerTest inherits test methods from
>> TransformerTest with that practice :)
>> I'll add more unit tests in addition to that.
>>
>>> Down the road we do have some kind of typeawareness with the rest-dsl
>>> where you can also specify in and output types which the swagger
>>> api-doc uses. And it does some automatic binding to/from
>>> pojo/xml/json. So we may have some overlap here, and may ponder if
>>> there is something we can eventually make use the new typeware stuff
>>> or meet in the middle or something. However maybe its better to take a
>>> look at that after the other stuff is more ready.
>>
>>
>> That sounds like a thing I need to look into, at least need to understand
>> what is available right now. Hopefully there's a good meeting point with it.
>>
>>> Also in Java DSL you may want to allow to specify a java type using
>>> the Class as type instead of having to type the FQN in a String. But
>>> mind in XML DSL this is not possible where you specify a String
>>> attribute for the type.
>>
>>
>> Good idea, I'll add that too!
>>
>>>> Thanks,
>>>> Tomo
>>>>
>>>>
>>>> On 09/17/2016 10:20 PM, Tomohisa Igarashi wrote:
>>>>>
>>>>>
>>>>> Hi Claus,
>>>>>
>>>>> Thanks for the reply. Sure that's fine, I agree 3.0 would be better to
>>>>> be
>>>>> targeted than 2.x as this introduces some schema updates.
>>>>>
>>>>> Including this one, I'm always looking for the chance to make any
>>>>> contribution to Camel. If there's anything I can help please let me
>>>>> know.
>>>>>
>>>>> Thanks,
>>>>> Tomo
>>>>>
>>>>> On 09/17/2016 06:33 PM, Claus Ibsen wrote:
>>>>>>
>>>>>>
>>>>>> Hey
>>>>>>
>>>>>> Can we take this discussion post Camel 2.18 release.
>>>>>>
>>>>>> We are working on the last details to get it ready, and its our main
>>>>>> focus to get this new release out.
>>>>>>
>>>>>> After this release we will pickup talks about the next releases
>>>>>> whether that is 2.19 or 3.0, and for the latter what the broad goals
>>>>>> of that is. What you talk about seems more of a 3.0 candidate to me,
>>>>>> than on 2.x.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 16, 2016 at 6:50 AM, Tomohisa Igarashi
>>>>>> <tm...@gmail.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi Camel developers,
>>>>>>>
>>>>>>> I'd like to propose an enhancement on handling data types of Camel
>>>>>>> message
>>>>>>> contents. To start a smooth discussion I implemented the idea first:
>>>>>>>
>>>>>>> https://github.com/igarashitm/camel/tree/contract-based-type-awareness
>>>>>>>
>>>>>>> And these testcases demonstrates the declarative transformer usage
>>>>>>> according
>>>>>>> to the declared data types:
>>>>>>> [Java DSL]
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-c14a7e8e88a6e41492946e1537bfb1cf
>>>>>>> [Spring DSL]
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-b2506c84ddde91438fc6374039e21534
>>>>>>>
>>>>>>> This adds input/output content type declaration on from and all other
>>>>>>> processors. It also introduces well-known Exchange properties,
>>>>>>> INPUT_TYPE
>>>>>>> and OUTPUT_TYPE which are used to specify the current message content
>>>>>>> type.
>>>>>>> The data type is URN like string starts with scheme, like
>>>>>>> java:org.example.ItemA or xml:{org.example.xml}ItemA.
>>>>>>>
>>>>>>> If the content type is declared via inputType/outputType/contract,
>>>>>>> the
>>>>>>> ContractProcessor wraps the actual processor and process
>>>>>>> transformation/validation according to the type, say if INPUT_TYPE
>>>>>>> exchange
>>>>>>> property has xml:{org.example.xml}ItemA and the declared inputType is
>>>>>>> xml:{org.example.xml}ItemB, then it transforms
>>>>>>> xml:{org.example.xml}ItemA
>>>>>>> content into xml:{org.example.xml}ItemB. The <transformers> element
>>>>>>> which is
>>>>>>> introduced right under the <camelContext> is the one to declare the
>>>>>>> mappings
>>>>>>> between transformer implementation and those from/to data type. I
>>>>>>> implemented only transformer first, but validator would be brought in
>>>>>>> in
>>>>>>> a
>>>>>>> same way. This way allows users to make data types visible in the
>>>>>>> route
>>>>>>> definition and keep the transformation/validation apart from route
>>>>>>> definition itself.
>>>>>>>
>>>>>>> The most important thing is that the ContractProcessor is involved
>>>>>>> only
>>>>>>> when
>>>>>>> content type is explicitly declared in a route definition, so that it
>>>>>>> never
>>>>>>> breaks existing camel routes. Ofcourse programatic
>>>>>>> transformations/validations we're doing today are still fully
>>>>>>> available.
>>>>>>> It's purely an addition to the existing camel features.
>>>>>>>
>>>>>>> Any thoughts? Does it sound acceptable to get merged into camel? ANY
>>>>>>> feedback would be really appreciated!
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Tomo
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>>
>>>
>
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2