You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Faraaz Sareshwala <fs...@quantcast.com> on 2016/01/26 03:06:14 UTC
Introducing myself and adding QFS
Hello everyone in the Bigtop community!
My name is Faraaz Sareshwala and I am an engineer working at Quantcast to integrate QFS (https://quantcast.github.io/qfs) into the Bigtop project. I just wanted to send a small note out to introduce myself to the community and announce that we are working on an integration. Feel free to reach out to me if you have any questions about QFS, our integration, or Quantcast itself.
At this point, I am nearly complete integrating QFS. I plan to submit a patch through github and JIRA later in the week or early next week once I get a chance to test everything is working correctly. I’ve read the wiki on contribution to the project, but if you guys have any other tips or guidance for me, I’d really appreciate that.
Patch coming soon!
Take care,
Faraaz
Re: Introducing myself and adding QFS
Posted by Konstantin Boudnik <co...@apache.org>.
What Evans said. Incremental steps are always preferable to one bulky patch.
This https://issues.apache.org/jira/browse/BIGTOP-1769 comes to mind as a
decent example of such addition. Basically, the following parts need to be
addresses for a new component:
- packaging
- package tests
- smoke tests (at very least). In case of QFS perhaps existing HCFS tests
might suffice
- puppet recipes for automatic deployment and configuration
- documentations/READMEs if any needed.
- license clearance: as you work toward adding new component make sure to run
'gradle rat' to make sure all new stuff is compliant with ASF license
requirements. If you add code licenses under different licenses, those
would need to be listed in the NOTICE
On Tue, Jan 26, 2016 at 10:50AM, Evans Ye wrote:
> Hi Faraaz Sareshwala,
>
> Welcome to the Bigtop community!
> If you're already doing the patch. I strongly suggest you to open up a
> JIRA, elaborate the integration, and discuss with the community first.
> That way we can minimal the effort of rewriting patches.
>
> One question after I took a glance on QFS, with RAID 1 setup on metaserver,
> the data can be persist safely, but what about the software running on it?
> Does that has SPOF?
Just a side node: as you know literally _all_ components in Hadoop-based stack
have SPOF. Best case sceneario - two SPOFs in case of HDFS HA. The only
exclusion of the rule is Ignite with its master-less architecture.
Cos
> Evans
>
>
> 2016-01-26 10:06 GMT+08:00 Faraaz Sareshwala <fs...@quantcast.com>:
>
> > Hello everyone in the Bigtop community!
> >
> > My name is Faraaz Sareshwala and I am an engineer working at Quantcast to
> > integrate QFS (https://quantcast.github.io/qfs) into the Bigtop project.
> > I just wanted to send a small note out to introduce myself to the community
> > and announce that we are working on an integration. Feel free to reach out
> > to me if you have any questions about QFS, our integration, or Quantcast
> > itself.
> >
> > At this point, I am nearly complete integrating QFS. I plan to submit a
> > patch through github and JIRA later in the week or early next week once I
> > get a chance to test everything is working correctly. I’ve read the wiki on
> > contribution to the project, but if you guys have any other tips or
> > guidance for me, I’d really appreciate that.
> >
> > Patch coming soon!
> >
> > Take care,
> > Faraaz
> >
> >
Re: Introducing myself and adding QFS
Posted by Evans Ye <ev...@apache.org>.
Hi Faraaz Sareshwala,
Welcome to the Bigtop community!
If you're already doing the patch. I strongly suggest you to open up a
JIRA, elaborate the integration, and discuss with the community first.
That way we can minimal the effort of rewriting patches.
One question after I took a glance on QFS, with RAID 1 setup on metaserver,
the data can be persist safely, but what about the software running on it?
Does that has SPOF?
Evans
2016-01-26 10:06 GMT+08:00 Faraaz Sareshwala <fs...@quantcast.com>:
> Hello everyone in the Bigtop community!
>
> My name is Faraaz Sareshwala and I am an engineer working at Quantcast to
> integrate QFS (https://quantcast.github.io/qfs) into the Bigtop project.
> I just wanted to send a small note out to introduce myself to the community
> and announce that we are working on an integration. Feel free to reach out
> to me if you have any questions about QFS, our integration, or Quantcast
> itself.
>
> At this point, I am nearly complete integrating QFS. I plan to submit a
> patch through github and JIRA later in the week or early next week once I
> get a chance to test everything is working correctly. I’ve read the wiki on
> contribution to the project, but if you guys have any other tips or
> guidance for me, I’d really appreciate that.
>
> Patch coming soon!
>
> Take care,
> Faraaz
>
>
Re: Introducing myself and adding QFS
Posted by jay vyas <ja...@gmail.com>.
Please add Your name to the MAINTAINERS.txt file in the PR for "QFS" !
On Tue, Jan 26, 2016 at 2:24 PM, Konstantin Boudnik <co...@apache.org> wrote:
> On Wed, Jan 27, 2016 at 01:05AM, Evans Ye wrote:
> > Oh, if I made you uncomfortable, you know I didn't mean that :)
>
> Oh believe me - it takes a lot more than that to make me uncomfortable ;)
> Besides, you gave me an opportunity to speak up about the SPOFs in the
> Hadoop
> ecosystem. Which I can do for hours!
>
> > I just want to know more about the design because I'm always interested
> in
> > the architecture things, especially when I see something different.
> > Even Spark just added HA in their master in recent release last year. So
> > it's totally fine.
> >
> > And the following is just for the discussion:
> >
> >> Just a side node: as you know literally _all_ components in Hadoop-based
> >> stack have SPOF.
> >
> > I think it's more related to the maturity of a component, but somehow it
> > will get there to eliminate the SPOF just like what HDFS, YARN, Oozie,
> Spark
> > did. For operations, two SPOFs is good enough for them to have a good
> sleep
> > at night ;)
>
> It might be ok for Oozie, but not for a file system. Because once your
> master
> has failed over to the standby you don't know how long that one has. It
> might
> crash in 5 months or in 2 minutes. That's why if you're an admin for a
> production critical cluster you'll get paged immediately. Perhaps 3
> o'clock in
> the morning.
>
> I spent last three years working on the active-active software for mission
> critical systems, so I would love to share some architectural thoughts with
> you, if you're interested.
>
> Cos
>
> > 2016-01-26 12:08 GMT+08:00 Jay Vyas <ja...@gmail.com>:
> >
> > > Awesome! It's great to hear about another HDFS alternative :)
> > >
> > > , We tried pretty hard to make bigtop more "HCFS" compliant, by adding
> the
> > > init hcfs tooling and so on, but never completed making the puppet
> recipes
> > > add support for other file systems.
> > >
> > >
> > > Would be very interesting to see if the quant cast team can take over
> on
> > > the HCFS integration so that bigtop is a home for more Than just one
> under
> > > filesystem .
> > >
> > > Let us know what your planning on working on, looking forward to it...!
> > >
> > >
> > >
> > >
> > > > On Jan 25, 2016, at 9:06 PM, Faraaz Sareshwala <
> > > fsareshwala@quantcast.com> wrote:
> > > >
> > > > Hello everyone in the Bigtop community!
> > > >
> > > > My name is Faraaz Sareshwala and I am an engineer working at
> Quantcast
> > > to integrate QFS (https://quantcast.github.io/qfs) into the Bigtop
> > > project. I just wanted to send a small note out to introduce myself to
> the
> > > community and announce that we are working on an integration. Feel
> free to
> > > reach out to me if you have any questions about QFS, our integration,
> or
> > > Quantcast itself.
> > > >
> > > > At this point, I am nearly complete integrating QFS. I plan to
> submit a
> > > patch through github and JIRA later in the week or early next week
> once I
> > > get a chance to test everything is working correctly. I’ve read the
> wiki on
> > > contribution to the project, but if you guys have any other tips or
> > > guidance for me, I’d really appreciate that.
> > > >
> > > > Patch coming soon!
> > > >
> > > > Take care,
> > > > Faraaz
> > > >
> > >
>
--
jay vyas
Re: Introducing myself and adding QFS
Posted by Konstantin Boudnik <co...@apache.org>.
On Wed, Jan 27, 2016 at 01:05AM, Evans Ye wrote:
> Oh, if I made you uncomfortable, you know I didn't mean that :)
Oh believe me - it takes a lot more than that to make me uncomfortable ;)
Besides, you gave me an opportunity to speak up about the SPOFs in the Hadoop
ecosystem. Which I can do for hours!
> I just want to know more about the design because I'm always interested in
> the architecture things, especially when I see something different.
> Even Spark just added HA in their master in recent release last year. So
> it's totally fine.
>
> And the following is just for the discussion:
>
>> Just a side node: as you know literally _all_ components in Hadoop-based
>> stack have SPOF.
>
> I think it's more related to the maturity of a component, but somehow it
> will get there to eliminate the SPOF just like what HDFS, YARN, Oozie, Spark
> did. For operations, two SPOFs is good enough for them to have a good sleep
> at night ;)
It might be ok for Oozie, but not for a file system. Because once your master
has failed over to the standby you don't know how long that one has. It might
crash in 5 months or in 2 minutes. That's why if you're an admin for a
production critical cluster you'll get paged immediately. Perhaps 3 o'clock in
the morning.
I spent last three years working on the active-active software for mission
critical systems, so I would love to share some architectural thoughts with
you, if you're interested.
Cos
> 2016-01-26 12:08 GMT+08:00 Jay Vyas <ja...@gmail.com>:
>
> > Awesome! It's great to hear about another HDFS alternative :)
> >
> > , We tried pretty hard to make bigtop more "HCFS" compliant, by adding the
> > init hcfs tooling and so on, but never completed making the puppet recipes
> > add support for other file systems.
> >
> >
> > Would be very interesting to see if the quant cast team can take over on
> > the HCFS integration so that bigtop is a home for more Than just one under
> > filesystem .
> >
> > Let us know what your planning on working on, looking forward to it...!
> >
> >
> >
> >
> > > On Jan 25, 2016, at 9:06 PM, Faraaz Sareshwala <
> > fsareshwala@quantcast.com> wrote:
> > >
> > > Hello everyone in the Bigtop community!
> > >
> > > My name is Faraaz Sareshwala and I am an engineer working at Quantcast
> > to integrate QFS (https://quantcast.github.io/qfs) into the Bigtop
> > project. I just wanted to send a small note out to introduce myself to the
> > community and announce that we are working on an integration. Feel free to
> > reach out to me if you have any questions about QFS, our integration, or
> > Quantcast itself.
> > >
> > > At this point, I am nearly complete integrating QFS. I plan to submit a
> > patch through github and JIRA later in the week or early next week once I
> > get a chance to test everything is working correctly. I’ve read the wiki on
> > contribution to the project, but if you guys have any other tips or
> > guidance for me, I’d really appreciate that.
> > >
> > > Patch coming soon!
> > >
> > > Take care,
> > > Faraaz
> > >
> >
Re: Introducing myself and adding QFS
Posted by Evans Ye <ev...@apache.org>.
Oh, if I made you uncomfortable, you know I didn't mean that :)
I just want to know more about the design because I'm always interested in
the architecture things, especially when I see something different.
Even Spark just added HA in their master in recent release last year. So
it's totally fine.
And the following is just for the discussion:
Just a side node: as you know literally _all_ components in Hadoop-based
stack
have SPOF.
I think it's more related to the maturity of a component, but somehow it
will get there to eliminate the SPOF just like what HDFS, YARN, Oozie,
Spark did.
For operations, two SPOFs is good enough for them to have a good sleep at
night ;)
2016-01-26 12:08 GMT+08:00 Jay Vyas <ja...@gmail.com>:
> Awesome! It's great to hear about another HDFS alternative :)
>
> , We tried pretty hard to make bigtop more "HCFS" compliant, by adding the
> init hcfs tooling and so on, but never completed making the puppet recipes
> add support for other file systems.
>
>
> Would be very interesting to see if the quant cast team can take over on
> the HCFS integration so that bigtop is a home for more Than just one under
> filesystem .
>
> Let us know what your planning on working on, looking forward to it...!
>
>
>
>
> > On Jan 25, 2016, at 9:06 PM, Faraaz Sareshwala <
> fsareshwala@quantcast.com> wrote:
> >
> > Hello everyone in the Bigtop community!
> >
> > My name is Faraaz Sareshwala and I am an engineer working at Quantcast
> to integrate QFS (https://quantcast.github.io/qfs) into the Bigtop
> project. I just wanted to send a small note out to introduce myself to the
> community and announce that we are working on an integration. Feel free to
> reach out to me if you have any questions about QFS, our integration, or
> Quantcast itself.
> >
> > At this point, I am nearly complete integrating QFS. I plan to submit a
> patch through github and JIRA later in the week or early next week once I
> get a chance to test everything is working correctly. I’ve read the wiki on
> contribution to the project, but if you guys have any other tips or
> guidance for me, I’d really appreciate that.
> >
> > Patch coming soon!
> >
> > Take care,
> > Faraaz
> >
>
Re: Introducing myself and adding QFS
Posted by Jay Vyas <ja...@gmail.com>.
Awesome! It's great to hear about another HDFS alternative :)
, We tried pretty hard to make bigtop more "HCFS" compliant, by adding the init hcfs tooling and so on, but never completed making the puppet recipes add support for other file systems.
Would be very interesting to see if the quant cast team can take over on the HCFS integration so that bigtop is a home for more Than just one under filesystem .
Let us know what your planning on working on, looking forward to it...!
> On Jan 25, 2016, at 9:06 PM, Faraaz Sareshwala <fs...@quantcast.com> wrote:
>
> Hello everyone in the Bigtop community!
>
> My name is Faraaz Sareshwala and I am an engineer working at Quantcast to integrate QFS (https://quantcast.github.io/qfs) into the Bigtop project. I just wanted to send a small note out to introduce myself to the community and announce that we are working on an integration. Feel free to reach out to me if you have any questions about QFS, our integration, or Quantcast itself.
>
> At this point, I am nearly complete integrating QFS. I plan to submit a patch through github and JIRA later in the week or early next week once I get a chance to test everything is working correctly. I’ve read the wiki on contribution to the project, but if you guys have any other tips or guidance for me, I’d really appreciate that.
>
> Patch coming soon!
>
> Take care,
> Faraaz
>