JDBC Poller implementation

[Priyanka Gugale <pr...@datatorrent.com>]

Hi,

JDBCPollerInputOperator in malhar repository emits comma separated list of
values as result of scan. As most of our input operators emit POJOs I am
planning to add an implementation which emits pojo.

I would like to discuss, if we should have two independent jdbc poll input
operators, one emits csv and other which emits pojo or we should have one
operator having two ports?

I prefer two operators to define clear intent of each operator, but if
anyone has different opinion please suggest.

-Priyanka

Re: JDBC Poller implementation

[Ashwin Chandra Putta <as...@gmail.com>]

+1 for common abstract class and specific implementations that are use case
specific.

Regards,
Ashwin.

On Mon, Aug 8, 2016 at 7:55 AM, Devendra Tagare <de...@datatorrent.com>
wrote:

> + 1 for a common Abstract Class.
>
> 0 - for adding ports.
>
> Use-case for CSV - data migration from columnar stores.Some users may want
> to move TB's of data from a store like Greenplum and then do
> transformations using the CSV parser that is already present.
>
> Thanks,
> Dev
>
> On Mon, Aug 8, 2016 at 7:39 AM, Yogi Devendra <yo...@apache.org>
> wrote:
>
> > +1 for Tushar's idea for common abstract class.
> >
> > Additionally, POJO based output should be considered for active
> > development. CSV output can be deprecated. Since it can be achieved
> easily
> > using POJO + CSV formatter.
> >
> > Operators are meant to be lego blocks for reusable functionality to
> achieve
> > higher level functionality by clubbing multiple operators together.
> >
> >
> > ~ Yogi
> >
> > On 8 August 2016 at 17:45, Tushar Gosavi <tu...@datatorrent.com> wrote:
> >
> > > I would prefer a common abstract class having emitTuple method. And
> > > two different implementation one for emitting comma separated values
> > > and other emitting pojo.
> > >
> > > Regards,
> > > -Tushar.
> > >
> > >
> > > On Mon, Aug 8, 2016 at 5:34 PM, Priyanka Gugale
> > > <pr...@datatorrent.com> wrote:
> > > > The concrete implementation is supposed to do mainly formatting of
> > input
> > > > data and emit in required form. Also it would be tricky to implement
> > some
> > > > abstract methods like "getTuple", you have to conditionally return
> the
> > > > values.
> > > >
> > > > -Priyanka
> > > >
> > > > On Mon, Aug 8, 2016 at 4:49 PM, Bhupesh Chawda <bh...@apache.org>
> > > wrote:
> > > >
> > > >> I am +1 for having two ports in the same concrete implementation.
> > > >> One port for POJOs and second one for CSV strings (if this is
> actually
> > > >> needed)
> > > >>
> > > >> +0 on having multiple concrete operators.
> > > >> I don't think it is necessary to create multiple concrete
> > > implementations
> > > >> just because we want the same data in different formats. Ports
> should
> > > serve
> > > >> the purpose.
> > > >>
> > > >> ~ Bhupesh
> > > >>
> > > >>
> > > >> On Mon, Aug 8, 2016 at 4:46 PM, Priyanka Gugale <
> > > priyanka@datatorrent.com>
> > > >> wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > JDBCPollerInputOperator in malhar repository emits comma separated
> > > list
> > > >> of
> > > >> > values as result of scan. As most of our input operators emit
> POJOs
> > I
> > > am
> > > >> > planning to add an implementation which emits pojo.
> > > >> >
> > > >> > I would like to discuss, if we should have two independent jdbc
> poll
> > > >> input
> > > >> > operators, one emits csv and other which emits pojo or we should
> > have
> > > one
> > > >> > operator having two ports?
> > > >> >
> > > >> > I prefer two operators to define clear intent of each operator,
> but
> > if
> > > >> > anyone has different opinion please suggest.
> > > >> >
> > > >> > -Priyanka
> > > >> >
> > > >>
> > >
> >
>



-- 

Regards,
Ashwin.

Re: JDBC Poller implementation

[Devendra Tagare <de...@datatorrent.com>]

+ 1 for a common Abstract Class.

0 - for adding ports.

Use-case for CSV - data migration from columnar stores.Some users may want
to move TB's of data from a store like Greenplum and then do
transformations using the CSV parser that is already present.

Thanks,
Dev

On Mon, Aug 8, 2016 at 7:39 AM, Yogi Devendra <yo...@apache.org>
wrote:

> +1 for Tushar's idea for common abstract class.
>
> Additionally, POJO based output should be considered for active
> development. CSV output can be deprecated. Since it can be achieved easily
> using POJO + CSV formatter.
>
> Operators are meant to be lego blocks for reusable functionality to achieve
> higher level functionality by clubbing multiple operators together.
>
>
> ~ Yogi
>
> On 8 August 2016 at 17:45, Tushar Gosavi <tu...@datatorrent.com> wrote:
>
> > I would prefer a common abstract class having emitTuple method. And
> > two different implementation one for emitting comma separated values
> > and other emitting pojo.
> >
> > Regards,
> > -Tushar.
> >
> >
> > On Mon, Aug 8, 2016 at 5:34 PM, Priyanka Gugale
> > <pr...@datatorrent.com> wrote:
> > > The concrete implementation is supposed to do mainly formatting of
> input
> > > data and emit in required form. Also it would be tricky to implement
> some
> > > abstract methods like "getTuple", you have to conditionally return the
> > > values.
> > >
> > > -Priyanka
> > >
> > > On Mon, Aug 8, 2016 at 4:49 PM, Bhupesh Chawda <bh...@apache.org>
> > wrote:
> > >
> > >> I am +1 for having two ports in the same concrete implementation.
> > >> One port for POJOs and second one for CSV strings (if this is actually
> > >> needed)
> > >>
> > >> +0 on having multiple concrete operators.
> > >> I don't think it is necessary to create multiple concrete
> > implementations
> > >> just because we want the same data in different formats. Ports should
> > serve
> > >> the purpose.
> > >>
> > >> ~ Bhupesh
> > >>
> > >>
> > >> On Mon, Aug 8, 2016 at 4:46 PM, Priyanka Gugale <
> > priyanka@datatorrent.com>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > JDBCPollerInputOperator in malhar repository emits comma separated
> > list
> > >> of
> > >> > values as result of scan. As most of our input operators emit POJOs
> I
> > am
> > >> > planning to add an implementation which emits pojo.
> > >> >
> > >> > I would like to discuss, if we should have two independent jdbc poll
> > >> input
> > >> > operators, one emits csv and other which emits pojo or we should
> have
> > one
> > >> > operator having two ports?
> > >> >
> > >> > I prefer two operators to define clear intent of each operator, but
> if
> > >> > anyone has different opinion please suggest.
> > >> >
> > >> > -Priyanka
> > >> >
> > >>
> >
>

Re: JDBC Poller implementation

[Yogi Devendra <yo...@apache.org>]

+1 for Tushar's idea for common abstract class.

Additionally, POJO based output should be considered for active
development. CSV output can be deprecated. Since it can be achieved easily
using POJO + CSV formatter.

Operators are meant to be lego blocks for reusable functionality to achieve
higher level functionality by clubbing multiple operators together.


~ Yogi

On 8 August 2016 at 17:45, Tushar Gosavi <tu...@datatorrent.com> wrote:

> I would prefer a common abstract class having emitTuple method. And
> two different implementation one for emitting comma separated values
> and other emitting pojo.
>
> Regards,
> -Tushar.
>
>
> On Mon, Aug 8, 2016 at 5:34 PM, Priyanka Gugale
> <pr...@datatorrent.com> wrote:
> > The concrete implementation is supposed to do mainly formatting of input
> > data and emit in required form. Also it would be tricky to implement some
> > abstract methods like "getTuple", you have to conditionally return the
> > values.
> >
> > -Priyanka
> >
> > On Mon, Aug 8, 2016 at 4:49 PM, Bhupesh Chawda <bh...@apache.org>
> wrote:
> >
> >> I am +1 for having two ports in the same concrete implementation.
> >> One port for POJOs and second one for CSV strings (if this is actually
> >> needed)
> >>
> >> +0 on having multiple concrete operators.
> >> I don't think it is necessary to create multiple concrete
> implementations
> >> just because we want the same data in different formats. Ports should
> serve
> >> the purpose.
> >>
> >> ~ Bhupesh
> >>
> >>
> >> On Mon, Aug 8, 2016 at 4:46 PM, Priyanka Gugale <
> priyanka@datatorrent.com>
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > JDBCPollerInputOperator in malhar repository emits comma separated
> list
> >> of
> >> > values as result of scan. As most of our input operators emit POJOs I
> am
> >> > planning to add an implementation which emits pojo.
> >> >
> >> > I would like to discuss, if we should have two independent jdbc poll
> >> input
> >> > operators, one emits csv and other which emits pojo or we should have
> one
> >> > operator having two ports?
> >> >
> >> > I prefer two operators to define clear intent of each operator, but if
> >> > anyone has different opinion please suggest.
> >> >
> >> > -Priyanka
> >> >
> >>
>

Re: JDBC Poller implementation

[Tushar Gosavi <tu...@datatorrent.com>]

I would prefer a common abstract class having emitTuple method. And
two different implementation one for emitting comma separated values
and other emitting pojo.

Regards,
-Tushar.


On Mon, Aug 8, 2016 at 5:34 PM, Priyanka Gugale
<pr...@datatorrent.com> wrote:
> The concrete implementation is supposed to do mainly formatting of input
> data and emit in required form. Also it would be tricky to implement some
> abstract methods like "getTuple", you have to conditionally return the
> values.
>
> -Priyanka
>
> On Mon, Aug 8, 2016 at 4:49 PM, Bhupesh Chawda <bh...@apache.org> wrote:
>
>> I am +1 for having two ports in the same concrete implementation.
>> One port for POJOs and second one for CSV strings (if this is actually
>> needed)
>>
>> +0 on having multiple concrete operators.
>> I don't think it is necessary to create multiple concrete implementations
>> just because we want the same data in different formats. Ports should serve
>> the purpose.
>>
>> ~ Bhupesh
>>
>>
>> On Mon, Aug 8, 2016 at 4:46 PM, Priyanka Gugale <pr...@datatorrent.com>
>> wrote:
>>
>> > Hi,
>> >
>> > JDBCPollerInputOperator in malhar repository emits comma separated list
>> of
>> > values as result of scan. As most of our input operators emit POJOs I am
>> > planning to add an implementation which emits pojo.
>> >
>> > I would like to discuss, if we should have two independent jdbc poll
>> input
>> > operators, one emits csv and other which emits pojo or we should have one
>> > operator having two ports?
>> >
>> > I prefer two operators to define clear intent of each operator, but if
>> > anyone has different opinion please suggest.
>> >
>> > -Priyanka
>> >
>>

Re: JDBC Poller implementation

[Priyanka Gugale <pr...@datatorrent.com>]

The concrete implementation is supposed to do mainly formatting of input
data and emit in required form. Also it would be tricky to implement some
abstract methods like "getTuple", you have to conditionally return the
values.

-Priyanka

On Mon, Aug 8, 2016 at 4:49 PM, Bhupesh Chawda <bh...@apache.org> wrote:

> I am +1 for having two ports in the same concrete implementation.
> One port for POJOs and second one for CSV strings (if this is actually
> needed)
>
> +0 on having multiple concrete operators.
> I don't think it is necessary to create multiple concrete implementations
> just because we want the same data in different formats. Ports should serve
> the purpose.
>
> ~ Bhupesh
>
>
> On Mon, Aug 8, 2016 at 4:46 PM, Priyanka Gugale <pr...@datatorrent.com>
> wrote:
>
> > Hi,
> >
> > JDBCPollerInputOperator in malhar repository emits comma separated list
> of
> > values as result of scan. As most of our input operators emit POJOs I am
> > planning to add an implementation which emits pojo.
> >
> > I would like to discuss, if we should have two independent jdbc poll
> input
> > operators, one emits csv and other which emits pojo or we should have one
> > operator having two ports?
> >
> > I prefer two operators to define clear intent of each operator, but if
> > anyone has different opinion please suggest.
> >
> > -Priyanka
> >
>

Re: JDBC Poller implementation

[Bhupesh Chawda <bh...@apache.org>]

I am +1 for having two ports in the same concrete implementation.
One port for POJOs and second one for CSV strings (if this is actually
needed)

+0 on having multiple concrete operators.
I don't think it is necessary to create multiple concrete implementations
just because we want the same data in different formats. Ports should serve
the purpose.

~ Bhupesh


On Mon, Aug 8, 2016 at 4:46 PM, Priyanka Gugale <pr...@datatorrent.com>
wrote:

> Hi,
>
> JDBCPollerInputOperator in malhar repository emits comma separated list of
> values as result of scan. As most of our input operators emit POJOs I am
> planning to add an implementation which emits pojo.
>
> I would like to discuss, if we should have two independent jdbc poll input
> operators, one emits csv and other which emits pojo or we should have one
> operator having two ports?
>
> I prefer two operators to define clear intent of each operator, but if
> anyone has different opinion please suggest.
>
> -Priyanka
>