You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by FO O <fo...@gmail.com> on 2020/03/22 00:14:37 UTC
Sequence of Transformers
Hi team!
My use case would benefit from running a SQL transformer followed by the
DMS transformer.
It seems my best options is to create a new transformer that is based on
the current DMS transformer and add the additional transformation I need
(add new columns, concatenate fields).
Wanted to see if there are additional recommendations that I should
consider instead of this one.
Thank you,
F
Re: Sequence of Transformers
Posted by James Walter <j1...@gmail.com>.
Thank you Shiyan.
Yes, for dev will be able to do that. Will figure out with the customer if
we can have an exception and do the same in prod.
Thank you again,
James
On Wed, Jul 15, 2020 at 1:45 PM Shiyan Xu <xu...@gmail.com>
wrote:
> hi James, glad that this could be helpful. Would it be possible for you to
> build the jar off master branch and use it in the meantime? You could
> install the jars to the EMR nodes during bootstrap.
>
> On Wed, Jul 15, 2020 at 8:11 AM James Walter <j1...@gmail.com>
> wrote:
>
> > Hello! +1 on this feature. I see that it is targeted to be shipped with
> > release 0.6.0. I know there are other priorities but if the community
> could
> > consider shipping this with earlier releases it would be great (my
> > customers don't like me messing up with what EMR provides by
> default....).
> >
> > Thank you, James.
> > On 2020/03/26 00:39:41, FO O <fo...@gmail.com> wrote:
> > > Thank you folks for the fast response and work.
> > >
> > > Vinoth Chandar <vi...@apache.org> escreveu no dia quarta, 25/03/2020
> > à(s)
> > > 11:22:
> > >
> > > > btw Raymond already has a PR up here for this :)
> > > > https://github.com/apache/incubator-hudi/pull/1440
> > > >
> > > > On Mon, Mar 23, 2020 at 5:32 PM Shiyan Xu <
> xu.shiyan.raymond@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Seems like an abstract class would be good enough for generic use?
> > > > > User can provide a list of `Transformer` then the abstract class
> just
> > > > apply
> > > > > all the way through the list.
> > > > > The implementation can be minimal for this approach.
> > > > >
> > > > > On Mon, Mar 23, 2020 at 4:12 PM Vinoth Chandar <vi...@apache.org>
> > > > wrote:
> > > > >
> > > > > > sg. Filed https://issues.apache.org/jira/browse/HUDI-731
> > > > > >
> > > > > > Someone looking to pick this? :). Its an nice feature to
> implement,
> > > > that
> > > > > > fits a good template..
> > > > > >
> > > > > > ofc we can discuss this more here in parallel
> > > > > >
> > > > > > On Mon, Mar 23, 2020 at 8:31 AM FO O <fo...@gmail.com>
> wrote:
> > > > > >
> > > > > > > Thank you Vinoth.
> > > > > > >
> > > > > > > >"If you are talking about implementing support for chained
> > calling
> > > > of
> > > > > > > multiple Transformers, within DeltaStreamer itself"
> > > > > > >
> > > > > > > Yes, chained calling support for transformers would be super
> > helpful,
> > > > > if
> > > > > > > this discussion can be revived it would be great.
> > > > > > >
> > > > > > > I see this useful for folks using DMS transformer and that need
> > some
> > > > > kind
> > > > > > > of transformation before the DMS transformer adds the op filed
> > for
> > > > > > initial
> > > > > > > load or when loading the CDC. In the meantime, I will create a
> > custom
> > > > > > > transformer.
> > > > > > >
> > > > > > > Thanks again,
> > > > > > > -F.
> > > > > > >
> > > > > > >
> > > > > > > Vinoth Chandar <vi...@apache.org> escreveu no dia domingo,
> > > > 22/03/2020
> > > > > > > à(s)
> > > > > > > 20:58:
> > > > > > >
> > > > > > > > Hi F,
> > > > > > > >
> > > > > > > > The Transformer interface allows you to basically plugin
> > anything
> > > > > that
> > > > > > > > takes a DataFrame and returns a transformed DataFrame. Does
> > that
> > > > > help?
> > > > > > > > If you are talking about implementing support for chained
> > calling
> > > > of
> > > > > > > > multiple Transformers, within DeltaStreamer itself..It has
> been
> > > > > > discussed
> > > > > > > > before.
> > > > > > > > And we can revive that conversation.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Vinoth
> > > > > > > >
> > > > > > > > On Sat, Mar 21, 2020 at 5:14 PM FO O <fo...@gmail.com>
> > wrote:
> > > > > > > >
> > > > > > > > > Hi team!
> > > > > > > > >
> > > > > > > > > My use case would benefit from running a SQL transformer
> > followed
> > > > > by
> > > > > > > the
> > > > > > > > > DMS transformer.
> > > > > > > > >
> > > > > > > > > It seems my best options is to create a new transformer
> that
> > is
> > > > > based
> > > > > > > on
> > > > > > > > > the current DMS transformer and add the additional
> > > > transformation I
> > > > > > > need
> > > > > > > > > (add new columns, concatenate fields).
> > > > > > > > >
> > > > > > > > > Wanted to see if there are additional recommendations that
> I
> > > > should
> > > > > > > > > consider instead of this one.
> > > > > > > > >
> > > > > > > > > Thank you,
> > > > > > > > > F
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Sequence of Transformers
Posted by Shiyan Xu <xu...@gmail.com>.
hi James, glad that this could be helpful. Would it be possible for you to
build the jar off master branch and use it in the meantime? You could
install the jars to the EMR nodes during bootstrap.
On Wed, Jul 15, 2020 at 8:11 AM James Walter <j1...@gmail.com>
wrote:
> Hello! +1 on this feature. I see that it is targeted to be shipped with
> release 0.6.0. I know there are other priorities but if the community could
> consider shipping this with earlier releases it would be great (my
> customers don't like me messing up with what EMR provides by default....).
>
> Thank you, James.
> On 2020/03/26 00:39:41, FO O <fo...@gmail.com> wrote:
> > Thank you folks for the fast response and work.
> >
> > Vinoth Chandar <vi...@apache.org> escreveu no dia quarta, 25/03/2020
> à(s)
> > 11:22:
> >
> > > btw Raymond already has a PR up here for this :)
> > > https://github.com/apache/incubator-hudi/pull/1440
> > >
> > > On Mon, Mar 23, 2020 at 5:32 PM Shiyan Xu <xu.shiyan.raymond@gmail.com
> >
> > > wrote:
> > >
> > > > Seems like an abstract class would be good enough for generic use?
> > > > User can provide a list of `Transformer` then the abstract class just
> > > apply
> > > > all the way through the list.
> > > > The implementation can be minimal for this approach.
> > > >
> > > > On Mon, Mar 23, 2020 at 4:12 PM Vinoth Chandar <vi...@apache.org>
> > > wrote:
> > > >
> > > > > sg. Filed https://issues.apache.org/jira/browse/HUDI-731
> > > > >
> > > > > Someone looking to pick this? :). Its an nice feature to implement,
> > > that
> > > > > fits a good template..
> > > > >
> > > > > ofc we can discuss this more here in parallel
> > > > >
> > > > > On Mon, Mar 23, 2020 at 8:31 AM FO O <fo...@gmail.com> wrote:
> > > > >
> > > > > > Thank you Vinoth.
> > > > > >
> > > > > > >"If you are talking about implementing support for chained
> calling
> > > of
> > > > > > multiple Transformers, within DeltaStreamer itself"
> > > > > >
> > > > > > Yes, chained calling support for transformers would be super
> helpful,
> > > > if
> > > > > > this discussion can be revived it would be great.
> > > > > >
> > > > > > I see this useful for folks using DMS transformer and that need
> some
> > > > kind
> > > > > > of transformation before the DMS transformer adds the op filed
> for
> > > > > initial
> > > > > > load or when loading the CDC. In the meantime, I will create a
> custom
> > > > > > transformer.
> > > > > >
> > > > > > Thanks again,
> > > > > > -F.
> > > > > >
> > > > > >
> > > > > > Vinoth Chandar <vi...@apache.org> escreveu no dia domingo,
> > > 22/03/2020
> > > > > > à(s)
> > > > > > 20:58:
> > > > > >
> > > > > > > Hi F,
> > > > > > >
> > > > > > > The Transformer interface allows you to basically plugin
> anything
> > > > that
> > > > > > > takes a DataFrame and returns a transformed DataFrame. Does
> that
> > > > help?
> > > > > > > If you are talking about implementing support for chained
> calling
> > > of
> > > > > > > multiple Transformers, within DeltaStreamer itself..It has been
> > > > > discussed
> > > > > > > before.
> > > > > > > And we can revive that conversation.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Vinoth
> > > > > > >
> > > > > > > On Sat, Mar 21, 2020 at 5:14 PM FO O <fo...@gmail.com>
> wrote:
> > > > > > >
> > > > > > > > Hi team!
> > > > > > > >
> > > > > > > > My use case would benefit from running a SQL transformer
> followed
> > > > by
> > > > > > the
> > > > > > > > DMS transformer.
> > > > > > > >
> > > > > > > > It seems my best options is to create a new transformer that
> is
> > > > based
> > > > > > on
> > > > > > > > the current DMS transformer and add the additional
> > > transformation I
> > > > > > need
> > > > > > > > (add new columns, concatenate fields).
> > > > > > > >
> > > > > > > > Wanted to see if there are additional recommendations that I
> > > should
> > > > > > > > consider instead of this one.
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > > F
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Sequence of Transformers
Posted by James Walter <j1...@gmail.com>.
Hello! +1 on this feature. I see that it is targeted to be shipped with release 0.6.0. I know there are other priorities but if the community could consider shipping this with earlier releases it would be great (my customers don't like me messing up with what EMR provides by default....).
Thank you, James.
On 2020/03/26 00:39:41, FO O <fo...@gmail.com> wrote:
> Thank you folks for the fast response and work.
>
> Vinoth Chandar <vi...@apache.org> escreveu no dia quarta, 25/03/2020 à(s)
> 11:22:
>
> > btw Raymond already has a PR up here for this :)
> > https://github.com/apache/incubator-hudi/pull/1440
> >
> > On Mon, Mar 23, 2020 at 5:32 PM Shiyan Xu <xu...@gmail.com>
> > wrote:
> >
> > > Seems like an abstract class would be good enough for generic use?
> > > User can provide a list of `Transformer` then the abstract class just
> > apply
> > > all the way through the list.
> > > The implementation can be minimal for this approach.
> > >
> > > On Mon, Mar 23, 2020 at 4:12 PM Vinoth Chandar <vi...@apache.org>
> > wrote:
> > >
> > > > sg. Filed https://issues.apache.org/jira/browse/HUDI-731
> > > >
> > > > Someone looking to pick this? :). Its an nice feature to implement,
> > that
> > > > fits a good template..
> > > >
> > > > ofc we can discuss this more here in parallel
> > > >
> > > > On Mon, Mar 23, 2020 at 8:31 AM FO O <fo...@gmail.com> wrote:
> > > >
> > > > > Thank you Vinoth.
> > > > >
> > > > > >"If you are talking about implementing support for chained calling
> > of
> > > > > multiple Transformers, within DeltaStreamer itself"
> > > > >
> > > > > Yes, chained calling support for transformers would be super helpful,
> > > if
> > > > > this discussion can be revived it would be great.
> > > > >
> > > > > I see this useful for folks using DMS transformer and that need some
> > > kind
> > > > > of transformation before the DMS transformer adds the op filed for
> > > > initial
> > > > > load or when loading the CDC. In the meantime, I will create a custom
> > > > > transformer.
> > > > >
> > > > > Thanks again,
> > > > > -F.
> > > > >
> > > > >
> > > > > Vinoth Chandar <vi...@apache.org> escreveu no dia domingo,
> > 22/03/2020
> > > > > à(s)
> > > > > 20:58:
> > > > >
> > > > > > Hi F,
> > > > > >
> > > > > > The Transformer interface allows you to basically plugin anything
> > > that
> > > > > > takes a DataFrame and returns a transformed DataFrame. Does that
> > > help?
> > > > > > If you are talking about implementing support for chained calling
> > of
> > > > > > multiple Transformers, within DeltaStreamer itself..It has been
> > > > discussed
> > > > > > before.
> > > > > > And we can revive that conversation.
> > > > > >
> > > > > > Thanks
> > > > > > Vinoth
> > > > > >
> > > > > > On Sat, Mar 21, 2020 at 5:14 PM FO O <fo...@gmail.com> wrote:
> > > > > >
> > > > > > > Hi team!
> > > > > > >
> > > > > > > My use case would benefit from running a SQL transformer followed
> > > by
> > > > > the
> > > > > > > DMS transformer.
> > > > > > >
> > > > > > > It seems my best options is to create a new transformer that is
> > > based
> > > > > on
> > > > > > > the current DMS transformer and add the additional
> > transformation I
> > > > > need
> > > > > > > (add new columns, concatenate fields).
> > > > > > >
> > > > > > > Wanted to see if there are additional recommendations that I
> > should
> > > > > > > consider instead of this one.
> > > > > > >
> > > > > > > Thank you,
> > > > > > > F
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Sequence of Transformers
Posted by FO O <fo...@gmail.com>.
Thank you folks for the fast response and work.
Vinoth Chandar <vi...@apache.org> escreveu no dia quarta, 25/03/2020 à(s)
11:22:
> btw Raymond already has a PR up here for this :)
> https://github.com/apache/incubator-hudi/pull/1440
>
> On Mon, Mar 23, 2020 at 5:32 PM Shiyan Xu <xu...@gmail.com>
> wrote:
>
> > Seems like an abstract class would be good enough for generic use?
> > User can provide a list of `Transformer` then the abstract class just
> apply
> > all the way through the list.
> > The implementation can be minimal for this approach.
> >
> > On Mon, Mar 23, 2020 at 4:12 PM Vinoth Chandar <vi...@apache.org>
> wrote:
> >
> > > sg. Filed https://issues.apache.org/jira/browse/HUDI-731
> > >
> > > Someone looking to pick this? :). Its an nice feature to implement,
> that
> > > fits a good template..
> > >
> > > ofc we can discuss this more here in parallel
> > >
> > > On Mon, Mar 23, 2020 at 8:31 AM FO O <fo...@gmail.com> wrote:
> > >
> > > > Thank you Vinoth.
> > > >
> > > > >"If you are talking about implementing support for chained calling
> of
> > > > multiple Transformers, within DeltaStreamer itself"
> > > >
> > > > Yes, chained calling support for transformers would be super helpful,
> > if
> > > > this discussion can be revived it would be great.
> > > >
> > > > I see this useful for folks using DMS transformer and that need some
> > kind
> > > > of transformation before the DMS transformer adds the op filed for
> > > initial
> > > > load or when loading the CDC. In the meantime, I will create a custom
> > > > transformer.
> > > >
> > > > Thanks again,
> > > > -F.
> > > >
> > > >
> > > > Vinoth Chandar <vi...@apache.org> escreveu no dia domingo,
> 22/03/2020
> > > > à(s)
> > > > 20:58:
> > > >
> > > > > Hi F,
> > > > >
> > > > > The Transformer interface allows you to basically plugin anything
> > that
> > > > > takes a DataFrame and returns a transformed DataFrame. Does that
> > help?
> > > > > If you are talking about implementing support for chained calling
> of
> > > > > multiple Transformers, within DeltaStreamer itself..It has been
> > > discussed
> > > > > before.
> > > > > And we can revive that conversation.
> > > > >
> > > > > Thanks
> > > > > Vinoth
> > > > >
> > > > > On Sat, Mar 21, 2020 at 5:14 PM FO O <fo...@gmail.com> wrote:
> > > > >
> > > > > > Hi team!
> > > > > >
> > > > > > My use case would benefit from running a SQL transformer followed
> > by
> > > > the
> > > > > > DMS transformer.
> > > > > >
> > > > > > It seems my best options is to create a new transformer that is
> > based
> > > > on
> > > > > > the current DMS transformer and add the additional
> transformation I
> > > > need
> > > > > > (add new columns, concatenate fields).
> > > > > >
> > > > > > Wanted to see if there are additional recommendations that I
> should
> > > > > > consider instead of this one.
> > > > > >
> > > > > > Thank you,
> > > > > > F
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Sequence of Transformers
Posted by Vinoth Chandar <vi...@apache.org>.
btw Raymond already has a PR up here for this :)
https://github.com/apache/incubator-hudi/pull/1440
On Mon, Mar 23, 2020 at 5:32 PM Shiyan Xu <xu...@gmail.com>
wrote:
> Seems like an abstract class would be good enough for generic use?
> User can provide a list of `Transformer` then the abstract class just apply
> all the way through the list.
> The implementation can be minimal for this approach.
>
> On Mon, Mar 23, 2020 at 4:12 PM Vinoth Chandar <vi...@apache.org> wrote:
>
> > sg. Filed https://issues.apache.org/jira/browse/HUDI-731
> >
> > Someone looking to pick this? :). Its an nice feature to implement, that
> > fits a good template..
> >
> > ofc we can discuss this more here in parallel
> >
> > On Mon, Mar 23, 2020 at 8:31 AM FO O <fo...@gmail.com> wrote:
> >
> > > Thank you Vinoth.
> > >
> > > >"If you are talking about implementing support for chained calling of
> > > multiple Transformers, within DeltaStreamer itself"
> > >
> > > Yes, chained calling support for transformers would be super helpful,
> if
> > > this discussion can be revived it would be great.
> > >
> > > I see this useful for folks using DMS transformer and that need some
> kind
> > > of transformation before the DMS transformer adds the op filed for
> > initial
> > > load or when loading the CDC. In the meantime, I will create a custom
> > > transformer.
> > >
> > > Thanks again,
> > > -F.
> > >
> > >
> > > Vinoth Chandar <vi...@apache.org> escreveu no dia domingo, 22/03/2020
> > > à(s)
> > > 20:58:
> > >
> > > > Hi F,
> > > >
> > > > The Transformer interface allows you to basically plugin anything
> that
> > > > takes a DataFrame and returns a transformed DataFrame. Does that
> help?
> > > > If you are talking about implementing support for chained calling of
> > > > multiple Transformers, within DeltaStreamer itself..It has been
> > discussed
> > > > before.
> > > > And we can revive that conversation.
> > > >
> > > > Thanks
> > > > Vinoth
> > > >
> > > > On Sat, Mar 21, 2020 at 5:14 PM FO O <fo...@gmail.com> wrote:
> > > >
> > > > > Hi team!
> > > > >
> > > > > My use case would benefit from running a SQL transformer followed
> by
> > > the
> > > > > DMS transformer.
> > > > >
> > > > > It seems my best options is to create a new transformer that is
> based
> > > on
> > > > > the current DMS transformer and add the additional transformation I
> > > need
> > > > > (add new columns, concatenate fields).
> > > > >
> > > > > Wanted to see if there are additional recommendations that I should
> > > > > consider instead of this one.
> > > > >
> > > > > Thank you,
> > > > > F
> > > > >
> > > >
> > >
> >
>
Re: Sequence of Transformers
Posted by Shiyan Xu <xu...@gmail.com>.
Seems like an abstract class would be good enough for generic use?
User can provide a list of `Transformer` then the abstract class just apply
all the way through the list.
The implementation can be minimal for this approach.
On Mon, Mar 23, 2020 at 4:12 PM Vinoth Chandar <vi...@apache.org> wrote:
> sg. Filed https://issues.apache.org/jira/browse/HUDI-731
>
> Someone looking to pick this? :). Its an nice feature to implement, that
> fits a good template..
>
> ofc we can discuss this more here in parallel
>
> On Mon, Mar 23, 2020 at 8:31 AM FO O <fo...@gmail.com> wrote:
>
> > Thank you Vinoth.
> >
> > >"If you are talking about implementing support for chained calling of
> > multiple Transformers, within DeltaStreamer itself"
> >
> > Yes, chained calling support for transformers would be super helpful, if
> > this discussion can be revived it would be great.
> >
> > I see this useful for folks using DMS transformer and that need some kind
> > of transformation before the DMS transformer adds the op filed for
> initial
> > load or when loading the CDC. In the meantime, I will create a custom
> > transformer.
> >
> > Thanks again,
> > -F.
> >
> >
> > Vinoth Chandar <vi...@apache.org> escreveu no dia domingo, 22/03/2020
> > à(s)
> > 20:58:
> >
> > > Hi F,
> > >
> > > The Transformer interface allows you to basically plugin anything that
> > > takes a DataFrame and returns a transformed DataFrame. Does that help?
> > > If you are talking about implementing support for chained calling of
> > > multiple Transformers, within DeltaStreamer itself..It has been
> discussed
> > > before.
> > > And we can revive that conversation.
> > >
> > > Thanks
> > > Vinoth
> > >
> > > On Sat, Mar 21, 2020 at 5:14 PM FO O <fo...@gmail.com> wrote:
> > >
> > > > Hi team!
> > > >
> > > > My use case would benefit from running a SQL transformer followed by
> > the
> > > > DMS transformer.
> > > >
> > > > It seems my best options is to create a new transformer that is based
> > on
> > > > the current DMS transformer and add the additional transformation I
> > need
> > > > (add new columns, concatenate fields).
> > > >
> > > > Wanted to see if there are additional recommendations that I should
> > > > consider instead of this one.
> > > >
> > > > Thank you,
> > > > F
> > > >
> > >
> >
>
Re: Sequence of Transformers
Posted by Vinoth Chandar <vi...@apache.org>.
sg. Filed https://issues.apache.org/jira/browse/HUDI-731
Someone looking to pick this? :). Its an nice feature to implement, that
fits a good template..
ofc we can discuss this more here in parallel
On Mon, Mar 23, 2020 at 8:31 AM FO O <fo...@gmail.com> wrote:
> Thank you Vinoth.
>
> >"If you are talking about implementing support for chained calling of
> multiple Transformers, within DeltaStreamer itself"
>
> Yes, chained calling support for transformers would be super helpful, if
> this discussion can be revived it would be great.
>
> I see this useful for folks using DMS transformer and that need some kind
> of transformation before the DMS transformer adds the op filed for initial
> load or when loading the CDC. In the meantime, I will create a custom
> transformer.
>
> Thanks again,
> -F.
>
>
> Vinoth Chandar <vi...@apache.org> escreveu no dia domingo, 22/03/2020
> à(s)
> 20:58:
>
> > Hi F,
> >
> > The Transformer interface allows you to basically plugin anything that
> > takes a DataFrame and returns a transformed DataFrame. Does that help?
> > If you are talking about implementing support for chained calling of
> > multiple Transformers, within DeltaStreamer itself..It has been discussed
> > before.
> > And we can revive that conversation.
> >
> > Thanks
> > Vinoth
> >
> > On Sat, Mar 21, 2020 at 5:14 PM FO O <fo...@gmail.com> wrote:
> >
> > > Hi team!
> > >
> > > My use case would benefit from running a SQL transformer followed by
> the
> > > DMS transformer.
> > >
> > > It seems my best options is to create a new transformer that is based
> on
> > > the current DMS transformer and add the additional transformation I
> need
> > > (add new columns, concatenate fields).
> > >
> > > Wanted to see if there are additional recommendations that I should
> > > consider instead of this one.
> > >
> > > Thank you,
> > > F
> > >
> >
>
Re: Sequence of Transformers
Posted by FO O <fo...@gmail.com>.
Thank you Vinoth.
>"If you are talking about implementing support for chained calling of
multiple Transformers, within DeltaStreamer itself"
Yes, chained calling support for transformers would be super helpful, if
this discussion can be revived it would be great.
I see this useful for folks using DMS transformer and that need some kind
of transformation before the DMS transformer adds the op filed for initial
load or when loading the CDC. In the meantime, I will create a custom
transformer.
Thanks again,
-F.
Vinoth Chandar <vi...@apache.org> escreveu no dia domingo, 22/03/2020 à(s)
20:58:
> Hi F,
>
> The Transformer interface allows you to basically plugin anything that
> takes a DataFrame and returns a transformed DataFrame. Does that help?
> If you are talking about implementing support for chained calling of
> multiple Transformers, within DeltaStreamer itself..It has been discussed
> before.
> And we can revive that conversation.
>
> Thanks
> Vinoth
>
> On Sat, Mar 21, 2020 at 5:14 PM FO O <fo...@gmail.com> wrote:
>
> > Hi team!
> >
> > My use case would benefit from running a SQL transformer followed by the
> > DMS transformer.
> >
> > It seems my best options is to create a new transformer that is based on
> > the current DMS transformer and add the additional transformation I need
> > (add new columns, concatenate fields).
> >
> > Wanted to see if there are additional recommendations that I should
> > consider instead of this one.
> >
> > Thank you,
> > F
> >
>
Re: Sequence of Transformers
Posted by Vinoth Chandar <vi...@apache.org>.
Hi F,
The Transformer interface allows you to basically plugin anything that
takes a DataFrame and returns a transformed DataFrame. Does that help?
If you are talking about implementing support for chained calling of
multiple Transformers, within DeltaStreamer itself..It has been discussed
before.
And we can revive that conversation.
Thanks
Vinoth
On Sat, Mar 21, 2020 at 5:14 PM FO O <fo...@gmail.com> wrote:
> Hi team!
>
> My use case would benefit from running a SQL transformer followed by the
> DMS transformer.
>
> It seems my best options is to create a new transformer that is based on
> the current DMS transformer and add the additional transformation I need
> (add new columns, concatenate fields).
>
> Wanted to see if there are additional recommendations that I should
> consider instead of this one.
>
> Thank you,
> F
>