You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by ananth <an...@gmail.com> on 2016/10/02 06:42:44 UTC
Kudu store operators
Hello All,
I was wondering if it would be worthwhile for the community to consider
support for Apache Kudu as a store ( as a contrib operator inside Apache
Malhar ) .
Here are some benefits I see:
1. Kudu is just declared 1.0 and has just been declared production ready.
2. Kudu as a store might a good a fit for many architectures in the
years to come because of its capabilities to provide mutability of
data ( unlike HDFS ) and optimized storage formats for scans.
3. It seems to also withstand high-throughput write patterns which
makes it a stable sink for Apex workflows which operate at very high
volumes.
Here are some links
* From the recent Strata conference
https://kudu.apache.org/2016/09/26/strata-nyc-kudu-talks.html
* https://kudu.apache.org/overview.html
I can implement this operator if the community feels it is worth adding
it to our code base. If so, could someone please assign the JIRA to me.
I have created this JIRA to track this :
https://issues.apache.org/jira/browse/APEXMALHAR-2278
Regards,
Ananth
Re: Kudu store operators
Posted by Mohit Jotwani <mo...@datatorrent.com>.
+1
Regards,
Mohit
On Mon, Oct 3, 2016 at 8:42 AM, Chaitanya Chebolu <chaitanya@datatorrent.com
> wrote:
> +1
>
> Regards,
> Chaitanya
>
> On Mon, Oct 3, 2016 at 6:01 PM, Sanjay Pujare <sa...@datatorrent.com>
> wrote:
>
> > +1
> >
> > On Oct 3, 2016 5:33 PM, "Sandeep Deshmukh" <sa...@datatorrent.com>
> > wrote:
> >
> > > +1
> > >
> > > Regards,
> > > Sandeep
> > >
> > > On Mon, Oct 3, 2016 at 10:16 AM, Tushar Gosavi <tushar@datatorrent.com
> >
> > > wrote:
> > >
> > > > +1, It will be great to have this operator.
> > > >
> > > > - Tushar.
> > > >
> > > > On Mon, Oct 3, 2016 at 8:15 AM, Chinmay Kolhatkar
> > > > <ch...@datatorrent.com> wrote:
> > > > > +1.
> > > > >
> > > > > - Chinmay.
> > > > >
> > > > > On 3 Oct 2016 7:25 a.m., "Amol Kekre" <am...@datatorrent.com>
> wrote:
> > > > >
> > > > >> Ananth,
> > > > >> This would be great to have. +1
> > > > >>
> > > > >> Thks
> > > > >> Amol
> > > > >>
> > > > >> On Sun, Oct 2, 2016 at 8:38 AM, Munagala Ramanath <
> > > ram@datatorrent.com>
> > > > >> wrote:
> > > > >>
> > > > >> > +1
> > > > >> >
> > > > >> > Kudu looks impressive from the overview, though it seems to
> still
> > be
> > > > >> > maturing.
> > > > >> >
> > > > >> > Ram
> > > > >> >
> > > > >> >
> > > > >> > On Sat, Oct 1, 2016 at 11:42 PM, ananth <ananthg.apex@gmail.com
> >
> > > > wrote:
> > > > >> >
> > > > >> > > Hello All,
> > > > >> > >
> > > > >> > > I was wondering if it would be worthwhile for the community to
> > > > consider
> > > > >> > > support for Apache Kudu as a store ( as a contrib operator
> > inside
> > > > >> Apache
> > > > >> > > Malhar ) .
> > > > >> > >
> > > > >> > > Here are some benefits I see:
> > > > >> > >
> > > > >> > > 1. Kudu is just declared 1.0 and has just been declared
> > production
> > > > >> ready.
> > > > >> > > 2. Kudu as a store might a good a fit for many architectures
> in
> > > the
> > > > >> > > years to come because of its capabilities to provide
> > mutability
> > > > of
> > > > >> > > data ( unlike HDFS ) and optimized storage formats for
> scans.
> > > > >> > > 3. It seems to also withstand high-throughput write patterns
> > which
> > > > >> > > makes it a stable sink for Apex workflows which operate at
> > very
> > > > high
> > > > >> > > volumes.
> > > > >> > >
> > > > >> > >
> > > > >> > > Here are some links
> > > > >> > >
> > > > >> > > * From the recent Strata conference
> > > > >> > > https://kudu.apache.org/2016/09/26/strata-nyc-kudu-talks.
> > html
> > > > >> > > * https://kudu.apache.org/overview.html
> > > > >> > >
> > > > >> > > I can implement this operator if the community feels it is
> worth
> > > > adding
> > > > >> > it
> > > > >> > > to our code base. If so, could someone please assign the JIRA
> to
> > > > me. I
> > > > >> > have
> > > > >> > > created this JIRA to track this :
> > https://issues.apache.org/jira
> > > > >> > > /browse/APEXMALHAR-2278
> > > > >> > >
> > > > >> > >
> > > > >> > > Regards,
> > > > >> > >
> > > > >> > > Ananth
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
Re: Kudu store operators
Posted by Chaitanya Chebolu <ch...@datatorrent.com>.
+1
Regards,
Chaitanya
On Mon, Oct 3, 2016 at 6:01 PM, Sanjay Pujare <sa...@datatorrent.com>
wrote:
> +1
>
> On Oct 3, 2016 5:33 PM, "Sandeep Deshmukh" <sa...@datatorrent.com>
> wrote:
>
> > +1
> >
> > Regards,
> > Sandeep
> >
> > On Mon, Oct 3, 2016 at 10:16 AM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > > +1, It will be great to have this operator.
> > >
> > > - Tushar.
> > >
> > > On Mon, Oct 3, 2016 at 8:15 AM, Chinmay Kolhatkar
> > > <ch...@datatorrent.com> wrote:
> > > > +1.
> > > >
> > > > - Chinmay.
> > > >
> > > > On 3 Oct 2016 7:25 a.m., "Amol Kekre" <am...@datatorrent.com> wrote:
> > > >
> > > >> Ananth,
> > > >> This would be great to have. +1
> > > >>
> > > >> Thks
> > > >> Amol
> > > >>
> > > >> On Sun, Oct 2, 2016 at 8:38 AM, Munagala Ramanath <
> > ram@datatorrent.com>
> > > >> wrote:
> > > >>
> > > >> > +1
> > > >> >
> > > >> > Kudu looks impressive from the overview, though it seems to still
> be
> > > >> > maturing.
> > > >> >
> > > >> > Ram
> > > >> >
> > > >> >
> > > >> > On Sat, Oct 1, 2016 at 11:42 PM, ananth <an...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > Hello All,
> > > >> > >
> > > >> > > I was wondering if it would be worthwhile for the community to
> > > consider
> > > >> > > support for Apache Kudu as a store ( as a contrib operator
> inside
> > > >> Apache
> > > >> > > Malhar ) .
> > > >> > >
> > > >> > > Here are some benefits I see:
> > > >> > >
> > > >> > > 1. Kudu is just declared 1.0 and has just been declared
> production
> > > >> ready.
> > > >> > > 2. Kudu as a store might a good a fit for many architectures in
> > the
> > > >> > > years to come because of its capabilities to provide
> mutability
> > > of
> > > >> > > data ( unlike HDFS ) and optimized storage formats for scans.
> > > >> > > 3. It seems to also withstand high-throughput write patterns
> which
> > > >> > > makes it a stable sink for Apex workflows which operate at
> very
> > > high
> > > >> > > volumes.
> > > >> > >
> > > >> > >
> > > >> > > Here are some links
> > > >> > >
> > > >> > > * From the recent Strata conference
> > > >> > > https://kudu.apache.org/2016/09/26/strata-nyc-kudu-talks.
> html
> > > >> > > * https://kudu.apache.org/overview.html
> > > >> > >
> > > >> > > I can implement this operator if the community feels it is worth
> > > adding
> > > >> > it
> > > >> > > to our code base. If so, could someone please assign the JIRA to
> > > me. I
> > > >> > have
> > > >> > > created this JIRA to track this :
> https://issues.apache.org/jira
> > > >> > > /browse/APEXMALHAR-2278
> > > >> > >
> > > >> > >
> > > >> > > Regards,
> > > >> > >
> > > >> > > Ananth
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
Re: Kudu store operators
Posted by Sanjay Pujare <sa...@datatorrent.com>.
+1
On Oct 3, 2016 5:33 PM, "Sandeep Deshmukh" <sa...@datatorrent.com> wrote:
> +1
>
> Regards,
> Sandeep
>
> On Mon, Oct 3, 2016 at 10:16 AM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > +1, It will be great to have this operator.
> >
> > - Tushar.
> >
> > On Mon, Oct 3, 2016 at 8:15 AM, Chinmay Kolhatkar
> > <ch...@datatorrent.com> wrote:
> > > +1.
> > >
> > > - Chinmay.
> > >
> > > On 3 Oct 2016 7:25 a.m., "Amol Kekre" <am...@datatorrent.com> wrote:
> > >
> > >> Ananth,
> > >> This would be great to have. +1
> > >>
> > >> Thks
> > >> Amol
> > >>
> > >> On Sun, Oct 2, 2016 at 8:38 AM, Munagala Ramanath <
> ram@datatorrent.com>
> > >> wrote:
> > >>
> > >> > +1
> > >> >
> > >> > Kudu looks impressive from the overview, though it seems to still be
> > >> > maturing.
> > >> >
> > >> > Ram
> > >> >
> > >> >
> > >> > On Sat, Oct 1, 2016 at 11:42 PM, ananth <an...@gmail.com>
> > wrote:
> > >> >
> > >> > > Hello All,
> > >> > >
> > >> > > I was wondering if it would be worthwhile for the community to
> > consider
> > >> > > support for Apache Kudu as a store ( as a contrib operator inside
> > >> Apache
> > >> > > Malhar ) .
> > >> > >
> > >> > > Here are some benefits I see:
> > >> > >
> > >> > > 1. Kudu is just declared 1.0 and has just been declared production
> > >> ready.
> > >> > > 2. Kudu as a store might a good a fit for many architectures in
> the
> > >> > > years to come because of its capabilities to provide mutability
> > of
> > >> > > data ( unlike HDFS ) and optimized storage formats for scans.
> > >> > > 3. It seems to also withstand high-throughput write patterns which
> > >> > > makes it a stable sink for Apex workflows which operate at very
> > high
> > >> > > volumes.
> > >> > >
> > >> > >
> > >> > > Here are some links
> > >> > >
> > >> > > * From the recent Strata conference
> > >> > > https://kudu.apache.org/2016/09/26/strata-nyc-kudu-talks.html
> > >> > > * https://kudu.apache.org/overview.html
> > >> > >
> > >> > > I can implement this operator if the community feels it is worth
> > adding
> > >> > it
> > >> > > to our code base. If so, could someone please assign the JIRA to
> > me. I
> > >> > have
> > >> > > created this JIRA to track this : https://issues.apache.org/jira
> > >> > > /browse/APEXMALHAR-2278
> > >> > >
> > >> > >
> > >> > > Regards,
> > >> > >
> > >> > > Ananth
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
Re: Kudu store operators
Posted by Sandeep Deshmukh <sa...@datatorrent.com>.
+1
Regards,
Sandeep
On Mon, Oct 3, 2016 at 10:16 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> +1, It will be great to have this operator.
>
> - Tushar.
>
> On Mon, Oct 3, 2016 at 8:15 AM, Chinmay Kolhatkar
> <ch...@datatorrent.com> wrote:
> > +1.
> >
> > - Chinmay.
> >
> > On 3 Oct 2016 7:25 a.m., "Amol Kekre" <am...@datatorrent.com> wrote:
> >
> >> Ananth,
> >> This would be great to have. +1
> >>
> >> Thks
> >> Amol
> >>
> >> On Sun, Oct 2, 2016 at 8:38 AM, Munagala Ramanath <ra...@datatorrent.com>
> >> wrote:
> >>
> >> > +1
> >> >
> >> > Kudu looks impressive from the overview, though it seems to still be
> >> > maturing.
> >> >
> >> > Ram
> >> >
> >> >
> >> > On Sat, Oct 1, 2016 at 11:42 PM, ananth <an...@gmail.com>
> wrote:
> >> >
> >> > > Hello All,
> >> > >
> >> > > I was wondering if it would be worthwhile for the community to
> consider
> >> > > support for Apache Kudu as a store ( as a contrib operator inside
> >> Apache
> >> > > Malhar ) .
> >> > >
> >> > > Here are some benefits I see:
> >> > >
> >> > > 1. Kudu is just declared 1.0 and has just been declared production
> >> ready.
> >> > > 2. Kudu as a store might a good a fit for many architectures in the
> >> > > years to come because of its capabilities to provide mutability
> of
> >> > > data ( unlike HDFS ) and optimized storage formats for scans.
> >> > > 3. It seems to also withstand high-throughput write patterns which
> >> > > makes it a stable sink for Apex workflows which operate at very
> high
> >> > > volumes.
> >> > >
> >> > >
> >> > > Here are some links
> >> > >
> >> > > * From the recent Strata conference
> >> > > https://kudu.apache.org/2016/09/26/strata-nyc-kudu-talks.html
> >> > > * https://kudu.apache.org/overview.html
> >> > >
> >> > > I can implement this operator if the community feels it is worth
> adding
> >> > it
> >> > > to our code base. If so, could someone please assign the JIRA to
> me. I
> >> > have
> >> > > created this JIRA to track this : https://issues.apache.org/jira
> >> > > /browse/APEXMALHAR-2278
> >> > >
> >> > >
> >> > > Regards,
> >> > >
> >> > > Ananth
> >> > >
> >> > >
> >> >
> >>
>
Re: Kudu store operators
Posted by Tushar Gosavi <tu...@datatorrent.com>.
+1, It will be great to have this operator.
- Tushar.
On Mon, Oct 3, 2016 at 8:15 AM, Chinmay Kolhatkar
<ch...@datatorrent.com> wrote:
> +1.
>
> - Chinmay.
>
> On 3 Oct 2016 7:25 a.m., "Amol Kekre" <am...@datatorrent.com> wrote:
>
>> Ananth,
>> This would be great to have. +1
>>
>> Thks
>> Amol
>>
>> On Sun, Oct 2, 2016 at 8:38 AM, Munagala Ramanath <ra...@datatorrent.com>
>> wrote:
>>
>> > +1
>> >
>> > Kudu looks impressive from the overview, though it seems to still be
>> > maturing.
>> >
>> > Ram
>> >
>> >
>> > On Sat, Oct 1, 2016 at 11:42 PM, ananth <an...@gmail.com> wrote:
>> >
>> > > Hello All,
>> > >
>> > > I was wondering if it would be worthwhile for the community to consider
>> > > support for Apache Kudu as a store ( as a contrib operator inside
>> Apache
>> > > Malhar ) .
>> > >
>> > > Here are some benefits I see:
>> > >
>> > > 1. Kudu is just declared 1.0 and has just been declared production
>> ready.
>> > > 2. Kudu as a store might a good a fit for many architectures in the
>> > > years to come because of its capabilities to provide mutability of
>> > > data ( unlike HDFS ) and optimized storage formats for scans.
>> > > 3. It seems to also withstand high-throughput write patterns which
>> > > makes it a stable sink for Apex workflows which operate at very high
>> > > volumes.
>> > >
>> > >
>> > > Here are some links
>> > >
>> > > * From the recent Strata conference
>> > > https://kudu.apache.org/2016/09/26/strata-nyc-kudu-talks.html
>> > > * https://kudu.apache.org/overview.html
>> > >
>> > > I can implement this operator if the community feels it is worth adding
>> > it
>> > > to our code base. If so, could someone please assign the JIRA to me. I
>> > have
>> > > created this JIRA to track this : https://issues.apache.org/jira
>> > > /browse/APEXMALHAR-2278
>> > >
>> > >
>> > > Regards,
>> > >
>> > > Ananth
>> > >
>> > >
>> >
>>
Re: Kudu store operators
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
+1.
- Chinmay.
On 3 Oct 2016 7:25 a.m., "Amol Kekre" <am...@datatorrent.com> wrote:
> Ananth,
> This would be great to have. +1
>
> Thks
> Amol
>
> On Sun, Oct 2, 2016 at 8:38 AM, Munagala Ramanath <ra...@datatorrent.com>
> wrote:
>
> > +1
> >
> > Kudu looks impressive from the overview, though it seems to still be
> > maturing.
> >
> > Ram
> >
> >
> > On Sat, Oct 1, 2016 at 11:42 PM, ananth <an...@gmail.com> wrote:
> >
> > > Hello All,
> > >
> > > I was wondering if it would be worthwhile for the community to consider
> > > support for Apache Kudu as a store ( as a contrib operator inside
> Apache
> > > Malhar ) .
> > >
> > > Here are some benefits I see:
> > >
> > > 1. Kudu is just declared 1.0 and has just been declared production
> ready.
> > > 2. Kudu as a store might a good a fit for many architectures in the
> > > years to come because of its capabilities to provide mutability of
> > > data ( unlike HDFS ) and optimized storage formats for scans.
> > > 3. It seems to also withstand high-throughput write patterns which
> > > makes it a stable sink for Apex workflows which operate at very high
> > > volumes.
> > >
> > >
> > > Here are some links
> > >
> > > * From the recent Strata conference
> > > https://kudu.apache.org/2016/09/26/strata-nyc-kudu-talks.html
> > > * https://kudu.apache.org/overview.html
> > >
> > > I can implement this operator if the community feels it is worth adding
> > it
> > > to our code base. If so, could someone please assign the JIRA to me. I
> > have
> > > created this JIRA to track this : https://issues.apache.org/jira
> > > /browse/APEXMALHAR-2278
> > >
> > >
> > > Regards,
> > >
> > > Ananth
> > >
> > >
> >
>
Re: Kudu store operators
Posted by Amol Kekre <am...@datatorrent.com>.
Ananth,
This would be great to have. +1
Thks
Amol
On Sun, Oct 2, 2016 at 8:38 AM, Munagala Ramanath <ra...@datatorrent.com>
wrote:
> +1
>
> Kudu looks impressive from the overview, though it seems to still be
> maturing.
>
> Ram
>
>
> On Sat, Oct 1, 2016 at 11:42 PM, ananth <an...@gmail.com> wrote:
>
> > Hello All,
> >
> > I was wondering if it would be worthwhile for the community to consider
> > support for Apache Kudu as a store ( as a contrib operator inside Apache
> > Malhar ) .
> >
> > Here are some benefits I see:
> >
> > 1. Kudu is just declared 1.0 and has just been declared production ready.
> > 2. Kudu as a store might a good a fit for many architectures in the
> > years to come because of its capabilities to provide mutability of
> > data ( unlike HDFS ) and optimized storage formats for scans.
> > 3. It seems to also withstand high-throughput write patterns which
> > makes it a stable sink for Apex workflows which operate at very high
> > volumes.
> >
> >
> > Here are some links
> >
> > * From the recent Strata conference
> > https://kudu.apache.org/2016/09/26/strata-nyc-kudu-talks.html
> > * https://kudu.apache.org/overview.html
> >
> > I can implement this operator if the community feels it is worth adding
> it
> > to our code base. If so, could someone please assign the JIRA to me. I
> have
> > created this JIRA to track this : https://issues.apache.org/jira
> > /browse/APEXMALHAR-2278
> >
> >
> > Regards,
> >
> > Ananth
> >
> >
>
Re: Kudu store operators
Posted by Munagala Ramanath <ra...@datatorrent.com>.
+1
Kudu looks impressive from the overview, though it seems to still be
maturing.
Ram
On Sat, Oct 1, 2016 at 11:42 PM, ananth <an...@gmail.com> wrote:
> Hello All,
>
> I was wondering if it would be worthwhile for the community to consider
> support for Apache Kudu as a store ( as a contrib operator inside Apache
> Malhar ) .
>
> Here are some benefits I see:
>
> 1. Kudu is just declared 1.0 and has just been declared production ready.
> 2. Kudu as a store might a good a fit for many architectures in the
> years to come because of its capabilities to provide mutability of
> data ( unlike HDFS ) and optimized storage formats for scans.
> 3. It seems to also withstand high-throughput write patterns which
> makes it a stable sink for Apex workflows which operate at very high
> volumes.
>
>
> Here are some links
>
> * From the recent Strata conference
> https://kudu.apache.org/2016/09/26/strata-nyc-kudu-talks.html
> * https://kudu.apache.org/overview.html
>
> I can implement this operator if the community feels it is worth adding it
> to our code base. If so, could someone please assign the JIRA to me. I have
> created this JIRA to track this : https://issues.apache.org/jira
> /browse/APEXMALHAR-2278
>
>
> Regards,
>
> Ananth
>
>
Re: Kudu store operators
Posted by Thomas Weise <th...@apache.org>.
Hi Ananth,
It would be great to have support for Kudu. You could start by looking at
similar integrations like the Geode operators and storage agent for
reference.
Please also see the contribution guidelines:
http://apex.apache.org/contributing.html
http://apex.apache.org/malhar-contributing.html
Thanks,
Thomas
On Sat, Oct 1, 2016 at 11:42 PM, ananth <an...@gmail.com> wrote:
> Hello All,
>
> I was wondering if it would be worthwhile for the community to consider
> support for Apache Kudu as a store ( as a contrib operator inside Apache
> Malhar ) .
>
> Here are some benefits I see:
>
> 1. Kudu is just declared 1.0 and has just been declared production ready.
> 2. Kudu as a store might a good a fit for many architectures in the
> years to come because of its capabilities to provide mutability of
> data ( unlike HDFS ) and optimized storage formats for scans.
> 3. It seems to also withstand high-throughput write patterns which
> makes it a stable sink for Apex workflows which operate at very high
> volumes.
>
>
> Here are some links
>
> * From the recent Strata conference
> https://kudu.apache.org/2016/09/26/strata-nyc-kudu-talks.html
> * https://kudu.apache.org/overview.html
>
> I can implement this operator if the community feels it is worth adding it
> to our code base. If so, could someone please assign the JIRA to me. I have
> created this JIRA to track this : https://issues.apache.org/jira
> /browse/APEXMALHAR-2278
>
>
> Regards,
>
> Ananth
>
>