You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Rafal Korytkowski <rk...@redhat.com> on 2018/07/18 12:04:13 UTC
Re: Re: Teiid Camel component
Hey,
I'm pretty much done with a proof of concept, at which you can peek in this
commit
https://github.com/rkorytkowski/camel/commit/f66eea569963730032713c62f057a5685fea7089
.
I've basically extended the SQL component to support a new dataSources
parameter, where you can pass multiple data sources instead of one e.g.
.to("teiid:select url from projects inner join licenses on projects.license
= licenses.license where project = #?dataSources=dbProjects,dbLicenses")
https://github.com/rkorytkowski/camel/blob/TEIIDTOOLS-451/components/camel-teiid/src/test/java/org/apache/camel/component/sql/TeiidDataSourcesTest.java#L87-L90
Currently it's just for SQL data sources. Next I'm planning to add a way to
pass csv and Excel files.
It makes us wonder, if instead of introducing the new TEIID component,
would you approve this new parameter in the SQL component itself. The
feature would be more straightforward to discover
and use for people who know nothing about Teiid. The dependency on Teiid
would be optional and we would throw an exception, if Teiid was not on the
classpath, but the parameter was used.
Thoughts?
On Tue, Jun 26, 2018 at 11:57 AM Claus Ibsen <cl...@gmail.com> wrote:
> On Tue, Jun 19, 2018 at 12:44 PM, Andrea Cosentino
> <an...@yahoo.com.invalid> wrote:
> > Not necessary. We can eventually wrap non OSGi-fied dependencies.
> >
>
> Yes that is correct as Andrea say. There are some components that
> cannot work in OSGi or does not make sense.
> However if possible we like to offer Camel components on as many
> runtimes as possible. So the ServiceMix team
> sometimes create OSGi bundles for 3rd party JARs that they release.
>
> An alternative is the wrap mechanism but its not as good as a real OSGi
> bundle.
>
>
> > Inviato da Yahoo Mail su Android
> >
> > Il mar, 19 giu, 2018 alle 12:42, Rafal Korytkowski<
> rkorytkowski@redhat.com> ha scritto: Hi Claus,
> >
> > Thanks a lot for your guidance!
> >
> > I've got one follow up question. Is it a strict requirement for
> component's
> > dependencies to be OSGI-fied?
> >
> > On Mon, Jun 18, 2018 at 9:01 AM Claus Ibsen <cl...@gmail.com>
> wrote:
> >
> >> Hi Rafal
> >>
> >> Welcome to the Camel community.
> >> It sounds great with a camel-teiid component.
> >>
> >> Writing a Camel component is very open ended and you have a lot of
> freedom.
> >>
> >> In terms of how to embed Teiid engine then Camel has no restrictions.
> >> If there is one instance in the JVM, then we can let Camel
> >> auto-discover
> >> and use the instance in the camel-teiid component, so the end user do
> >> not have to explicit configure this wiring.
> >>
> >> Camel has its Registry abstraction over for example Spring Boot /
> >> ApplicationContext (bean registry) or CDI's BeanManager, or OSGi
> >> Service Registry etc.
> >> So what other components is doing is to lookup in this registry if
> >> there is a single instance of a specific class type, and then use
> >> that.
> >>
> >> There is a little guide here how to add a new Camel component
> >> http://camel.apache.org/add-new-component-guide.html
> >>
> >> We do have a maven archetype to create a new camel component project,
> >> or what other people sometimes do is to copy an existing component and
> >> then delete/modify its source code.
> >> http://camel.apache.org/camel-maven-archetypes.html
> >>
> >>
> >>
> >>
> >>
> >> On Fri, Jun 15, 2018 at 12:48 PM, Rafal Korytkowski
> >> <rk...@redhat.com> wrote:
> >> > Hi,
> >> >
> >> > We're considering implementing a Camel component for Teiid (
> >> http://teiid.io/)
> >> > and looking for your insights on some design aspects and general
> >> thoughts.
> >> >
> >> > Teiid is a data virtualization engine that comes as in-place real-time
> >> > integration of heterogeneous data sources (~50). Teiid exposes them
> >> through
> >> > a SQL engine and ODATA REST endpoints. It is typically run as a server
> >> > being accessed by SQL clients.
> >> >
> >> > Data sources supported by Teiid are pretty much a subset of what Camel
> >> > already supports, but the benefit we see is that it provides the SQL
> >> engine
> >> > to access them all in a unified way, which makes integrations more
> >> > straightforward than using custom APIs.
> >> >
> >> > It is currently possible to use the sql-camel/jdbc-camel component to
> >> > connect to a running Teiid instance, but we are looking for a tighter
> >> > integration by providing a way to embed the Teiid engine in a Camel
> >> > component. It would simplify its usage.
> >> >
> >> > Would you have any recommendations for writing such a component?
> >> >
> >> > One aspect we need to consider is, if it would be possible to reuse
> >> somehow
> >> > an instance of Teiid within a route or across routes or share some of
> >> > Teiid's metadata so that we do not have to bring up an instance each
> >> time.
> >> >
> >> > Finally, we have just started a similar conversion on the Teiid forum
> so,
> >> > if you are interested please also see
> >> > https://developer.jboss.org/thread/278138
> >> > --
> >> >
> >> > -Rafał
> >>
> >>
> >>
> >> --
> >> Claus Ibsen
> >> -----------------
> >> http://davsclaus.com @davsclaus
> >> Camel in Action 2: https://www.manning.com/ibsen2
> >>
> > --
> >
> > -Rafał
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
--
-Rafał
Re: Re: Teiid Camel component
Posted by Claus Ibsen <cl...@gmail.com>.
On Mon, Jul 23, 2018 at 1:55 PM, Hiram Chirino <hi...@hiramchirino.com> wrote:
> Hi!
>
> Nice work! I tend to think lets just do the new teiid component. It keep
> things more focused for things like the generated meta data for the SQL and
> Teiid components.
>
Yeah I also think its best to keep it in a new camel-teiid component.
You can still extend camel-sql, which we do for example in camel-elsql.
> On Wed, Jul 18, 2018 at 8:04 AM Rafal Korytkowski <rk...@redhat.com>
> wrote:
>
>> Hey,
>>
>> I'm pretty much done with a proof of concept, at which you can peek in this
>> commit
>>
>> https://github.com/rkorytkowski/camel/commit/f66eea569963730032713c62f057a5685fea7089
>> .
>>
>> I've basically extended the SQL component to support a new dataSources
>> parameter, where you can pass multiple data sources instead of one e.g.
>>
>> .to("teiid:select url from projects inner join licenses on projects.license
>> = licenses.license where project = #?dataSources=dbProjects,dbLicenses")
>>
>>
>> https://github.com/rkorytkowski/camel/blob/TEIIDTOOLS-451/components/camel-teiid/src/test/java/org/apache/camel/component/sql/TeiidDataSourcesTest.java#L87-L90
>>
>> Currently it's just for SQL data sources. Next I'm planning to add a way to
>> pass csv and Excel files.
>>
>> It makes us wonder, if instead of introducing the new TEIID component,
>> would you approve this new parameter in the SQL component itself. The
>> feature would be more straightforward to discover
>> and use for people who know nothing about Teiid. The dependency on Teiid
>> would be optional and we would throw an exception, if Teiid was not on the
>> classpath, but the parameter was used.
>>
>> Thoughts?
>>
>> On Tue, Jun 26, 2018 at 11:57 AM Claus Ibsen <cl...@gmail.com>
>> wrote:
>>
>> > On Tue, Jun 19, 2018 at 12:44 PM, Andrea Cosentino
>> > <an...@yahoo.com.invalid> wrote:
>> > > Not necessary. We can eventually wrap non OSGi-fied dependencies.
>> > >
>> >
>> > Yes that is correct as Andrea say. There are some components that
>> > cannot work in OSGi or does not make sense.
>> > However if possible we like to offer Camel components on as many
>> > runtimes as possible. So the ServiceMix team
>> > sometimes create OSGi bundles for 3rd party JARs that they release.
>> >
>> > An alternative is the wrap mechanism but its not as good as a real OSGi
>> > bundle.
>> >
>> >
>> > > Inviato da Yahoo Mail su Android
>> > >
>> > > Il mar, 19 giu, 2018 alle 12:42, Rafal Korytkowski<
>> > rkorytkowski@redhat.com> ha scritto: Hi Claus,
>> > >
>> > > Thanks a lot for your guidance!
>> > >
>> > > I've got one follow up question. Is it a strict requirement for
>> > component's
>> > > dependencies to be OSGI-fied?
>> > >
>> > > On Mon, Jun 18, 2018 at 9:01 AM Claus Ibsen <cl...@gmail.com>
>> > wrote:
>> > >
>> > >> Hi Rafal
>> > >>
>> > >> Welcome to the Camel community.
>> > >> It sounds great with a camel-teiid component.
>> > >>
>> > >> Writing a Camel component is very open ended and you have a lot of
>> > freedom.
>> > >>
>> > >> In terms of how to embed Teiid engine then Camel has no restrictions.
>> > >> If there is one instance in the JVM, then we can let Camel
>> > >> auto-discover
>> > >> and use the instance in the camel-teiid component, so the end user do
>> > >> not have to explicit configure this wiring.
>> > >>
>> > >> Camel has its Registry abstraction over for example Spring Boot /
>> > >> ApplicationContext (bean registry) or CDI's BeanManager, or OSGi
>> > >> Service Registry etc.
>> > >> So what other components is doing is to lookup in this registry if
>> > >> there is a single instance of a specific class type, and then use
>> > >> that.
>> > >>
>> > >> There is a little guide here how to add a new Camel component
>> > >> http://camel.apache.org/add-new-component-guide.html
>> > >>
>> > >> We do have a maven archetype to create a new camel component project,
>> > >> or what other people sometimes do is to copy an existing component and
>> > >> then delete/modify its source code.
>> > >> http://camel.apache.org/camel-maven-archetypes.html
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Fri, Jun 15, 2018 at 12:48 PM, Rafal Korytkowski
>> > >> <rk...@redhat.com> wrote:
>> > >> > Hi,
>> > >> >
>> > >> > We're considering implementing a Camel component for Teiid (
>> > >> http://teiid.io/)
>> > >> > and looking for your insights on some design aspects and general
>> > >> thoughts.
>> > >> >
>> > >> > Teiid is a data virtualization engine that comes as in-place
>> real-time
>> > >> > integration of heterogeneous data sources (~50). Teiid exposes them
>> > >> through
>> > >> > a SQL engine and ODATA REST endpoints. It is typically run as a
>> server
>> > >> > being accessed by SQL clients.
>> > >> >
>> > >> > Data sources supported by Teiid are pretty much a subset of what
>> Camel
>> > >> > already supports, but the benefit we see is that it provides the SQL
>> > >> engine
>> > >> > to access them all in a unified way, which makes integrations more
>> > >> > straightforward than using custom APIs.
>> > >> >
>> > >> > It is currently possible to use the sql-camel/jdbc-camel component
>> to
>> > >> > connect to a running Teiid instance, but we are looking for a
>> tighter
>> > >> > integration by providing a way to embed the Teiid engine in a Camel
>> > >> > component. It would simplify its usage.
>> > >> >
>> > >> > Would you have any recommendations for writing such a component?
>> > >> >
>> > >> > One aspect we need to consider is, if it would be possible to reuse
>> > >> somehow
>> > >> > an instance of Teiid within a route or across routes or share some
>> of
>> > >> > Teiid's metadata so that we do not have to bring up an instance each
>> > >> time.
>> > >> >
>> > >> > Finally, we have just started a similar conversion on the Teiid
>> forum
>> > so,
>> > >> > if you are interested please also see
>> > >> > https://developer.jboss.org/thread/278138
>> > >> > --
>> > >> >
>> > >> > -Rafał
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Claus Ibsen
>> > >> -----------------
>> > >> http://davsclaus.com @davsclaus
>> > >> Camel in Action 2: https://www.manning.com/ibsen2
>> > >>
>> > > --
>> > >
>> > > -Rafał
>> >
>> >
>> >
>> > --
>> > Claus Ibsen
>> > -----------------
>> > http://davsclaus.com @davsclaus
>> > Camel in Action 2: https://www.manning.com/ibsen2
>> >
>> --
>>
>> -Rafał
>>
>
>
> --
> Hiram Chirino
> Engineering | Red Hat, Inc.
> hchirino@redhat.com | fusesource.com | redhat.com
> skype: hiramchirino | twitter: @hiramchirino
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Re: Teiid Camel component
Posted by Hiram Chirino <hi...@hiramchirino.com>.
Hi!
Nice work! I tend to think lets just do the new teiid component. It keep
things more focused for things like the generated meta data for the SQL and
Teiid components.
On Wed, Jul 18, 2018 at 8:04 AM Rafal Korytkowski <rk...@redhat.com>
wrote:
> Hey,
>
> I'm pretty much done with a proof of concept, at which you can peek in this
> commit
>
> https://github.com/rkorytkowski/camel/commit/f66eea569963730032713c62f057a5685fea7089
> .
>
> I've basically extended the SQL component to support a new dataSources
> parameter, where you can pass multiple data sources instead of one e.g.
>
> .to("teiid:select url from projects inner join licenses on projects.license
> = licenses.license where project = #?dataSources=dbProjects,dbLicenses")
>
>
> https://github.com/rkorytkowski/camel/blob/TEIIDTOOLS-451/components/camel-teiid/src/test/java/org/apache/camel/component/sql/TeiidDataSourcesTest.java#L87-L90
>
> Currently it's just for SQL data sources. Next I'm planning to add a way to
> pass csv and Excel files.
>
> It makes us wonder, if instead of introducing the new TEIID component,
> would you approve this new parameter in the SQL component itself. The
> feature would be more straightforward to discover
> and use for people who know nothing about Teiid. The dependency on Teiid
> would be optional and we would throw an exception, if Teiid was not on the
> classpath, but the parameter was used.
>
> Thoughts?
>
> On Tue, Jun 26, 2018 at 11:57 AM Claus Ibsen <cl...@gmail.com>
> wrote:
>
> > On Tue, Jun 19, 2018 at 12:44 PM, Andrea Cosentino
> > <an...@yahoo.com.invalid> wrote:
> > > Not necessary. We can eventually wrap non OSGi-fied dependencies.
> > >
> >
> > Yes that is correct as Andrea say. There are some components that
> > cannot work in OSGi or does not make sense.
> > However if possible we like to offer Camel components on as many
> > runtimes as possible. So the ServiceMix team
> > sometimes create OSGi bundles for 3rd party JARs that they release.
> >
> > An alternative is the wrap mechanism but its not as good as a real OSGi
> > bundle.
> >
> >
> > > Inviato da Yahoo Mail su Android
> > >
> > > Il mar, 19 giu, 2018 alle 12:42, Rafal Korytkowski<
> > rkorytkowski@redhat.com> ha scritto: Hi Claus,
> > >
> > > Thanks a lot for your guidance!
> > >
> > > I've got one follow up question. Is it a strict requirement for
> > component's
> > > dependencies to be OSGI-fied?
> > >
> > > On Mon, Jun 18, 2018 at 9:01 AM Claus Ibsen <cl...@gmail.com>
> > wrote:
> > >
> > >> Hi Rafal
> > >>
> > >> Welcome to the Camel community.
> > >> It sounds great with a camel-teiid component.
> > >>
> > >> Writing a Camel component is very open ended and you have a lot of
> > freedom.
> > >>
> > >> In terms of how to embed Teiid engine then Camel has no restrictions.
> > >> If there is one instance in the JVM, then we can let Camel
> > >> auto-discover
> > >> and use the instance in the camel-teiid component, so the end user do
> > >> not have to explicit configure this wiring.
> > >>
> > >> Camel has its Registry abstraction over for example Spring Boot /
> > >> ApplicationContext (bean registry) or CDI's BeanManager, or OSGi
> > >> Service Registry etc.
> > >> So what other components is doing is to lookup in this registry if
> > >> there is a single instance of a specific class type, and then use
> > >> that.
> > >>
> > >> There is a little guide here how to add a new Camel component
> > >> http://camel.apache.org/add-new-component-guide.html
> > >>
> > >> We do have a maven archetype to create a new camel component project,
> > >> or what other people sometimes do is to copy an existing component and
> > >> then delete/modify its source code.
> > >> http://camel.apache.org/camel-maven-archetypes.html
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Fri, Jun 15, 2018 at 12:48 PM, Rafal Korytkowski
> > >> <rk...@redhat.com> wrote:
> > >> > Hi,
> > >> >
> > >> > We're considering implementing a Camel component for Teiid (
> > >> http://teiid.io/)
> > >> > and looking for your insights on some design aspects and general
> > >> thoughts.
> > >> >
> > >> > Teiid is a data virtualization engine that comes as in-place
> real-time
> > >> > integration of heterogeneous data sources (~50). Teiid exposes them
> > >> through
> > >> > a SQL engine and ODATA REST endpoints. It is typically run as a
> server
> > >> > being accessed by SQL clients.
> > >> >
> > >> > Data sources supported by Teiid are pretty much a subset of what
> Camel
> > >> > already supports, but the benefit we see is that it provides the SQL
> > >> engine
> > >> > to access them all in a unified way, which makes integrations more
> > >> > straightforward than using custom APIs.
> > >> >
> > >> > It is currently possible to use the sql-camel/jdbc-camel component
> to
> > >> > connect to a running Teiid instance, but we are looking for a
> tighter
> > >> > integration by providing a way to embed the Teiid engine in a Camel
> > >> > component. It would simplify its usage.
> > >> >
> > >> > Would you have any recommendations for writing such a component?
> > >> >
> > >> > One aspect we need to consider is, if it would be possible to reuse
> > >> somehow
> > >> > an instance of Teiid within a route or across routes or share some
> of
> > >> > Teiid's metadata so that we do not have to bring up an instance each
> > >> time.
> > >> >
> > >> > Finally, we have just started a similar conversion on the Teiid
> forum
> > so,
> > >> > if you are interested please also see
> > >> > https://developer.jboss.org/thread/278138
> > >> > --
> > >> >
> > >> > -Rafał
> > >>
> > >>
> > >>
> > >> --
> > >> Claus Ibsen
> > >> -----------------
> > >> http://davsclaus.com @davsclaus
> > >> Camel in Action 2: https://www.manning.com/ibsen2
> > >>
> > > --
> > >
> > > -Rafał
> >
> >
> >
> > --
> > Claus Ibsen
> > -----------------
> > http://davsclaus.com @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
> >
> --
>
> -Rafał
>
--
Hiram Chirino
Engineering | Red Hat, Inc.
hchirino@redhat.com | fusesource.com | redhat.com
skype: hiramchirino | twitter: @hiramchirino