You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by krishna rb <kr...@gmail.com> on 2017/03/11 10:02:38 UTC
Contributing to oodt Project
Hello everyone,
This is R B Krishna, studying 3rd year Computer Science at National
Institute of Technology, Trichy, India.
I am interested in contributing to the project.
Could someone suggest me some issues that I could take up as a newbie and
guide me through it?
I am currently going through the codebase and trying to get comfortable
with it.
Thanks,
R B Krishna.
Re: Contributing to oodt Project
Posted by krishna rb <kr...@gmail.com>.
Thank you, that was a great explanation about the project.
I'll start working on it and get a proposal ready for review soon.
Is there any issues that I can take up to contribute to the OODT project
for now?
I feel I'll get to know the codebase properly if I contribute some more
patches soon.
Thanks,
R B Krishna
On Sun, Mar 12, 2017 at 7:44 PM, Tom Barber <to...@meteorite.bi> wrote:
> Cool.
>
> We'll the Kafka idea is pretty simple, although I need to plan the
> implementation locations properly. So OODT is made up of a bunch of
> components, the most common ones being the Curator, the File Manager and
> the Workflow Manager.
>
> Currently tracking data through OODT isn't as simple as it could or perhaps
> should be, so the idea behind OODT-946 is to implement Kafka as a pub sub
> system for OODT. For example:
>
> The curator allows users to ingest data into the OODT platform via the file
> manager, so one implementation would be to replace the current
> communication between file manager and curator with a Kafka message bus. So
> user X curates some data and when they ingest it, instead of calling the
> file manager directly the curation message is pushed onto the Kafka message
> bus and a number of services could be subscribed to this. They may be the
> file manager which would pick it up and ingest the data, but it might also
> be an Audit component that tracks the flow of data through OODT, it might
> also be the workflow manager etc.
>
> Similarly when the file manager ingests data, instead of it simply calling
> the backend factory for Solr, Lucene or whatever to store the ingested
> data, it would be great if it again, pushed the message to Kafka, which in
> turn allowed different file manager backends to subscribe to the queue,
> perhaps to pick up specific ingestion file types or whatever. But also that
> push got optionally audited but the audit component also subscribed to the
> mesage queue, so you know when what file got ingested where, along with the
> entry into the catalog.
>
> That type of thing.
>
> On Sun, Mar 12, 2017 at 1:37 PM, krishna rb <kr...@gmail.com> wrote:
>
> > Yes sir, I am currently looking for issues so that I can understand the
> > project and contribute to it.
> > Also, I am interested in participating in Gsoc 2017 and I found the
> > project: Implement Kafka Messaging System Within OODT
> > <https://issues.apache.org/jira/browse/OODT-946> very interesting.
> > I am currently learning Apache Kafka.
> > It will be immensely helpful if you could explain about the idea.
> >
> > Thanks,
> > R B Krishna
> >
> > On Mar 12, 2017 6:56 PM, "Tom Barber" <to...@meteorite.bi> wrote:
> >
> > > Hi R B,
> > >
> > > You're a Java developer I assume?
> > >
> > > Thanks
> > >
> > > Tom
> > >
> > > On Sat, Mar 11, 2017 at 10:02 AM, krishna rb <kr...@gmail.com>
> > wrote:
> > >
> > > > Hello everyone,
> > > > This is R B Krishna, studying 3rd year Computer Science at National
> > > > Institute of Technology, Trichy, India.
> > > > I am interested in contributing to the project.
> > > >
> > > > Could someone suggest me some issues that I could take up as a newbie
> > and
> > > > guide me through it?
> > > >
> > > > I am currently going through the codebase and trying to get
> comfortable
> > > > with it.
> > > >
> > > > Thanks,
> > > > R B Krishna.
> > > >
> > >
> >
>
Re: Contributing to oodt Project
Posted by Tom Barber <to...@meteorite.bi>.
Cool.
We'll the Kafka idea is pretty simple, although I need to plan the
implementation locations properly. So OODT is made up of a bunch of
components, the most common ones being the Curator, the File Manager and
the Workflow Manager.
Currently tracking data through OODT isn't as simple as it could or perhaps
should be, so the idea behind OODT-946 is to implement Kafka as a pub sub
system for OODT. For example:
The curator allows users to ingest data into the OODT platform via the file
manager, so one implementation would be to replace the current
communication between file manager and curator with a Kafka message bus. So
user X curates some data and when they ingest it, instead of calling the
file manager directly the curation message is pushed onto the Kafka message
bus and a number of services could be subscribed to this. They may be the
file manager which would pick it up and ingest the data, but it might also
be an Audit component that tracks the flow of data through OODT, it might
also be the workflow manager etc.
Similarly when the file manager ingests data, instead of it simply calling
the backend factory for Solr, Lucene or whatever to store the ingested
data, it would be great if it again, pushed the message to Kafka, which in
turn allowed different file manager backends to subscribe to the queue,
perhaps to pick up specific ingestion file types or whatever. But also that
push got optionally audited but the audit component also subscribed to the
mesage queue, so you know when what file got ingested where, along with the
entry into the catalog.
That type of thing.
On Sun, Mar 12, 2017 at 1:37 PM, krishna rb <kr...@gmail.com> wrote:
> Yes sir, I am currently looking for issues so that I can understand the
> project and contribute to it.
> Also, I am interested in participating in Gsoc 2017 and I found the
> project: Implement Kafka Messaging System Within OODT
> <https://issues.apache.org/jira/browse/OODT-946> very interesting.
> I am currently learning Apache Kafka.
> It will be immensely helpful if you could explain about the idea.
>
> Thanks,
> R B Krishna
>
> On Mar 12, 2017 6:56 PM, "Tom Barber" <to...@meteorite.bi> wrote:
>
> > Hi R B,
> >
> > You're a Java developer I assume?
> >
> > Thanks
> >
> > Tom
> >
> > On Sat, Mar 11, 2017 at 10:02 AM, krishna rb <kr...@gmail.com>
> wrote:
> >
> > > Hello everyone,
> > > This is R B Krishna, studying 3rd year Computer Science at National
> > > Institute of Technology, Trichy, India.
> > > I am interested in contributing to the project.
> > >
> > > Could someone suggest me some issues that I could take up as a newbie
> and
> > > guide me through it?
> > >
> > > I am currently going through the codebase and trying to get comfortable
> > > with it.
> > >
> > > Thanks,
> > > R B Krishna.
> > >
> >
>
Re: Contributing to oodt Project
Posted by krishna rb <kr...@gmail.com>.
Yes sir, I am currently looking for issues so that I can understand the
project and contribute to it.
Also, I am interested in participating in Gsoc 2017 and I found the
project: Implement Kafka Messaging System Within OODT
<https://issues.apache.org/jira/browse/OODT-946> very interesting.
I am currently learning Apache Kafka.
It will be immensely helpful if you could explain about the idea.
Thanks,
R B Krishna
On Mar 12, 2017 6:56 PM, "Tom Barber" <to...@meteorite.bi> wrote:
> Hi R B,
>
> You're a Java developer I assume?
>
> Thanks
>
> Tom
>
> On Sat, Mar 11, 2017 at 10:02 AM, krishna rb <kr...@gmail.com> wrote:
>
> > Hello everyone,
> > This is R B Krishna, studying 3rd year Computer Science at National
> > Institute of Technology, Trichy, India.
> > I am interested in contributing to the project.
> >
> > Could someone suggest me some issues that I could take up as a newbie and
> > guide me through it?
> >
> > I am currently going through the codebase and trying to get comfortable
> > with it.
> >
> > Thanks,
> > R B Krishna.
> >
>
Re: Contributing to oodt Project
Posted by Tom Barber <to...@meteorite.bi>.
Hi R B,
You're a Java developer I assume?
Thanks
Tom
On Sat, Mar 11, 2017 at 10:02 AM, krishna rb <kr...@gmail.com> wrote:
> Hello everyone,
> This is R B Krishna, studying 3rd year Computer Science at National
> Institute of Technology, Trichy, India.
> I am interested in contributing to the project.
>
> Could someone suggest me some issues that I could take up as a newbie and
> guide me through it?
>
> I am currently going through the codebase and trying to get comfortable
> with it.
>
> Thanks,
> R B Krishna.
>