You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Andres Danter <ad...@gmail.com> on 2013/05/01 07:25:47 UTC
Re: GSoC proposal: Ambari for Accumulo - Final Draft
Hello Ambari and Accumulo developers,
Here is the link to the final draft of my proposal. Not too much has
changed. Some of the language on the previous draft was confusing, so I
cleaned that up, and I did add some Ambari tidbits. The biggest addition,
in my opinion, is the table that lists the different versions of software I
will be using. Please take a look at that table to ensure I don't have a
mismatch. I have a nagging feeling that the version of Hadoop that Ambari
can deploy is newer than the version Accumulo 1.4.3 is comfortable with.
Thank you all again for your support and feedback. Here is the link:
https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
Andres
On Mon, Apr 29, 2013 at 8:03 PM, Andres Danter <ad...@gmail.com> wrote:
> Hi Drew,
>
> Sorry for replying so late. Unfortunately, this project is a solo thing.
> Hopefully we can team up on another project in the future.
>
> Thanks for the support.
>
> Andres
>
>
> On Fri, Apr 26, 2013 at 11:01 AM, Drew Pierce <dr...@live.com> wrote:
>
>> Not sure if these are solo initiatives or those needing assistance. Let
>> me know if it is the latter :)
>>
>>
>> > Date: Fri, 26 Apr 2013 10:47:21 -0400
>> > Subject: Re: GSoC proposal: Ambari for Accumulo - forever : )
>> > From: adanter@gmail.com
>> > To: dev@accumulo.apache.org
>> >
>> > Thanks, Drew, for the enthusiam.
>> >
>> > And thanks, John, for the good information. It is definitely an area I
>> > need to look into.
>> > On Apr 26, 2013 10:31 AM, "John Vines" <vi...@apache.org> wrote:
>> >
>> > > The init.d scripts we provide are well tested, provided you have
>> accumulo
>> > > installed into /usr/lib. There is probably room for improvement to
>> utilize
>> > > /etc/defaults/accumulo to make it configurable. This makes them not
>> > > configurable with the existing RPM packaging internal to Accumulo. And
>> > > Ambari uses the init.d scripts to start/stop things.
>> > >
>> > > As mentioned in the other thread, I'm pretty sure Ambari is pretty
>> tightly
>> > > integrated with Bigtop. There is an Accumulo ticket for bigtop
>> support (
>> > > Accumulo-138 ). I went ahead and threw a patch on there in which the
>> RPM
>> > > generated has been tested (the deb is lacking though). Furthermore,
>> they do
>> > > not use the given Accumulo init.d scripts, so there may be room for
>> > > improvement in there somewhere, but I'm not sure.
>> > >
>> > > Should you need to work down that path more, I can definitely provide
>> more
>> > > guidance.
>> > >
>> > >
>> > > On Thu, Apr 25, 2013 at 12:18 PM, Keith Turner <ke...@deenlo.com>
>> wrote:
>> > >
>> > > > On Thu, Apr 25, 2013 at 11:40 AM, Andres Danter <ad...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > Excellent feedback, Keith. Thanks.
>> > > > >
>> > > > > I should have mentioned the REST API in the proposal. I'm not
>> sure if
>> > > I
>> > > > > can use it for the CLI. I will certainly try, since I would like
>> to
>> > > > adhere
>> > > > > to the Ambari architecture as much as possible.
>> > > > >
>> > > > > As for your other points:
>> > > > >
>> > > > > *3.1.1 loggers do not exists in 1.5 ... so would only need to
>> consider
>> > > > them
>> > > > > if working w/ 1.4 . I suppose you would want to work Accumulo
>> 1.4,
>> > > since
>> > > > > that the current stable release.*
>> > > > >
>> > > > > Yes. I was going to work with Accumulo 1.4, but it is good to
>> know that
>> > > > the
>> > > > > loggers are not supported in 1.5.
>> > > > >
>> > > > > *3.1.3
>> > > > > Accumulo supports reading system config from xml local file and
>> then
>> > > > > zookeeper (whats set in zookeeper takes precedence). The nice
>> thing
>> > > > about
>> > > > > setting something a config zookeeper, is that process restart is
>> not
>> > > > > required.*
>> > > > >
>> > > > > Good to know this too. Does this only affect Zookeeper? I should
>> > > > > distinguish between processes that may need restarts to load a
>> > > > > configuration vs those that do not. Is the Zookeeper process the
>> only
>> > > > one
>> > > > > like that?
>> > > > >
>> > > > >
>> > > > Not sure about zookeeper, I think it requires a restart. Accumulo
>> can
>> > > > optionally store a lot of its configuration in zookeeper. When
>> Accumulo
>> > > > configuration stored in zookeeper is changed, Accumulo processes
>> may pick
>> > > > up these config changes immediately w/o restart. Some properties
>> do
>> > > > require a restart when changed in zookeeper. This is documented in
>> > > > Accumulo config documentation.
>> > > >
>> > > >
>> > > > >
>> > > > > *For section 3.2, are you proposing adding features to the
>> existing
>> > > > Ambari
>> > > > > GUI? If so, are the Accumulo specific?*
>> > > > >
>> > > > > I am proposing to add features to the existing Ambari GUI. The
>> idea
>> > > here
>> > > > > is that something like user access controls can be generalized so
>> that
>> > > > any
>> > > > > Hadoop component being managed by Ambari is able to utilize them.
>> I'm
>> > > > > hoping that the Ambari PMC is OK with me doing this. I do not
>> want to
>> > > > > introduce any Accumulo-specific features that would make no sense
>> for
>> > > > other
>> > > > > components.
>> > > > > *
>> > > > > It seems that Ambari relies on RPMs. Its possible you may have to
>> > > spend
>> > > > > time creating Accumulo RPMs that are useful for Ambari. You may
>> want to
>> > > > > think about allocating some time for this. Christopher and John
>> can
>> > > give
>> > > > > you more info about the current state of Accumulo RPMs.*
>> > > > >
>> > > > > This is a very good point. While building RPMs is not difficult,
>> it
>> > > does
>> > > > > require time to setup the configuration and test the builds. I
>> will
>> > > > > include time for this in the proposal.
>> > > > >
>> > > >
>> > > > I suppose I am really thinking about are scripts for starting and
>> > > stopping
>> > > > Accumulo. For an RPM to be useful it should provide good
>> /etc/init.d
>> > > > scripts for starting and stopping Accumulo processes. I am not
>> sure
>> > > what
>> > > > the status of these scripts is in the various RPMs built by
>> Accumulo.
>> > > This
>> > > > may not even be a concern, I am not sure how Ambari starts and stop
>> > > > services. If you have not already, its something to consider when
>> > > > scheduling.
>> > > >
>> > > >
>> > > > >
>> > > > > Thanks again, Keith.
>> > > > >
>> > > > > Andres
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Thu, Apr 25, 2013 at 10:54 AM, Keith Turner <ke...@deenlo.com>
>> > > wrote:
>> > > > >
>> > > > > > About the REST API, I meant to ask if this could be used from
>> the
>> > > > command
>> > > > > > line currently? Or is too cumbersome?
>> > > > > >
>> > > > > >
>> > > > > > On Thu, Apr 25, 2013 at 10:09 AM, Andres Danter <
>> adanter@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > > Yup. Looks like that's the way to go. Here is the link to
>> the
>> > > > > document.
>> > > > > > > Let me know if you have any problems accessing it.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
>> > > > > > >
>> > > > > > >
>> > > > > > > On Thu, Apr 25, 2013 at 9:51 AM, Keith Turner <
>> keith@deenlo.com>
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > Nothing. I suppose the mailing list is stripping it off.
>> Could
>> > > > post
>> > > > > > it
>> > > > > > > as
>> > > > > > > > an attachment to the ticket or put it on google drive or
>> dropbox
>> > > > and
>> > > > > > > send a
>> > > > > > > > link.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Thu, Apr 25, 2013 at 9:44 AM, Andres Danter <
>> > > adanter@gmail.com>
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Weird. I see the attachment on the email I sent, which
>> means
>> > > > that
>> > > > > it
>> > > > > > > > > might have been stripped off. Here it is again. Let me
>> know
>> > > if
>> > > > > you
>> > > > > > > > still
>> > > > > > > > > don't see it.
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > >
>> > > > > > > > > Andres
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Thu, Apr 25, 2013 at 9:18 AM, Keith Turner <
>> > > keith@deenlo.com>
>> > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> was there supposed to be an attachment?
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > > >> On Thu, Apr 25, 2013 at 2:19 AM, Andres Danter <
>> > > > adanter@gmail.com
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > > >>
>> > > > > > > > >> > Hi Billie,
>> > > > > > > > >> >
>> > > > > > > > >> > Here is a draft of my proposal. I've yet to write the
>> > > > technical
>> > > > > > > > >> approach
>> > > > > > > > >> > section (have not idea what I'm going to put there)
>> and the
>> > > > > "about
>> > > > > > > me"
>> > > > > > > > >> > section. I definitely appreciate any feedback from
>> you and
>> > > > > anyone
>> > > > > > > in
>> > > > > > > > >> the
>> > > > > > > > >> > development group.
>> > > > > > > > >> >
>> > > > > > > > >> > Thank you,
>> > > > > > > > >> >
>> > > > > > > > >> > Andres
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>>
>>
>
>
Re: GSoC proposal: Ambari for Accumulo - Final Draft
Posted by Billie Rinaldi <bi...@gmail.com>.
I think including it in the proposal isn't necessary. I just wanted to
give you a heads up. :-)
Billie
On Wed, May 1, 2013 at 7:28 AM, Andres Danter <ad...@gmail.com> wrote:
> Thanks, Billie. I agree about working with 1.5.0. I was going to be
> sneaky and try to integrate with both versions, but I didn't want to put it
> on the proposal in case I run into complications. Let me know if you want
> me to include that in the proposal, though, and I'll add it.
>
> Thanks again.
>
> Andres
>
>
> On Wed, May 1, 2013 at 10:20 AM, Billie Rinaldi <billie.rinaldi@gmail.com
> >wrote:
>
> > Andres,
> >
> > The proposal looks good. I believe Accumulo will work with Hadoop 1.0.4.
> > You may want to work both with Accumulo 1.5.0 and 1.4.3 if we get 1.5.0
> > released soon enough.
> >
> > Billie
> >
> >
> > On Tue, Apr 30, 2013 at 10:25 PM, Andres Danter <ad...@gmail.com>
> wrote:
> >
> > > Hello Ambari and Accumulo developers,
> > >
> > > Here is the link to the final draft of my proposal. Not too much has
> > > changed. Some of the language on the previous draft was confusing, so
> I
> > > cleaned that up, and I did add some Ambari tidbits. The biggest
> > addition,
> > > in my opinion, is the table that lists the different versions of
> > software I
> > > will be using. Please take a look at that table to ensure I don't
> have a
> > > mismatch. I have a nagging feeling that the version of Hadoop that
> > Ambari
> > > can deploy is newer than the version Accumulo 1.4.3 is comfortable
> with.
> > >
> > > Thank you all again for your support and feedback. Here is the link:
> > >
> > >
> > >
> >
> https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
> > >
> > >
> > > Andres
> > >
> > >
> > > On Mon, Apr 29, 2013 at 8:03 PM, Andres Danter <ad...@gmail.com>
> > wrote:
> > >
> > > > Hi Drew,
> > > >
> > > > Sorry for replying so late. Unfortunately, this project is a solo
> > thing.
> > > > Hopefully we can team up on another project in the future.
> > > >
> > > > Thanks for the support.
> > > >
> > > > Andres
> > > >
> > > >
> > > > On Fri, Apr 26, 2013 at 11:01 AM, Drew Pierce <dr...@live.com>
> > > wrote:
> > > >
> > > >> Not sure if these are solo initiatives or those needing assistance.
> > Let
> > > >> me know if it is the latter :)
> > > >>
> > > >>
> > > >> > Date: Fri, 26 Apr 2013 10:47:21 -0400
> > > >> > Subject: Re: GSoC proposal: Ambari for Accumulo - forever : )
> > > >> > From: adanter@gmail.com
> > > >> > To: dev@accumulo.apache.org
> > > >> >
> > > >> > Thanks, Drew, for the enthusiam.
> > > >> >
> > > >> > And thanks, John, for the good information. It is definitely an
> > area
> > > I
> > > >> > need to look into.
> > > >> > On Apr 26, 2013 10:31 AM, "John Vines" <vi...@apache.org> wrote:
> > > >> >
> > > >> > > The init.d scripts we provide are well tested, provided you have
> > > >> accumulo
> > > >> > > installed into /usr/lib. There is probably room for improvement
> to
> > > >> utilize
> > > >> > > /etc/defaults/accumulo to make it configurable. This makes them
> > not
> > > >> > > configurable with the existing RPM packaging internal to
> Accumulo.
> > > And
> > > >> > > Ambari uses the init.d scripts to start/stop things.
> > > >> > >
> > > >> > > As mentioned in the other thread, I'm pretty sure Ambari is
> pretty
> > > >> tightly
> > > >> > > integrated with Bigtop. There is an Accumulo ticket for bigtop
> > > >> support (
> > > >> > > Accumulo-138 ). I went ahead and threw a patch on there in which
> > the
> > > >> RPM
> > > >> > > generated has been tested (the deb is lacking though).
> > Furthermore,
> > > >> they do
> > > >> > > not use the given Accumulo init.d scripts, so there may be room
> > for
> > > >> > > improvement in there somewhere, but I'm not sure.
> > > >> > >
> > > >> > > Should you need to work down that path more, I can definitely
> > > provide
> > > >> more
> > > >> > > guidance.
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Apr 25, 2013 at 12:18 PM, Keith Turner <
> keith@deenlo.com>
> > > >> wrote:
> > > >> > >
> > > >> > > > On Thu, Apr 25, 2013 at 11:40 AM, Andres Danter <
> > > adanter@gmail.com>
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > Excellent feedback, Keith. Thanks.
> > > >> > > > >
> > > >> > > > > I should have mentioned the REST API in the proposal. I'm
> not
> > > >> sure if
> > > >> > > I
> > > >> > > > > can use it for the CLI. I will certainly try, since I would
> > > like
> > > >> to
> > > >> > > > adhere
> > > >> > > > > to the Ambari architecture as much as possible.
> > > >> > > > >
> > > >> > > > > As for your other points:
> > > >> > > > >
> > > >> > > > > *3.1.1 loggers do not exists in 1.5 ... so would only need
> to
> > > >> consider
> > > >> > > > them
> > > >> > > > > if working w/ 1.4 . I suppose you would want to work
> Accumulo
> > > >> 1.4,
> > > >> > > since
> > > >> > > > > that the current stable release.*
> > > >> > > > >
> > > >> > > > > Yes. I was going to work with Accumulo 1.4, but it is good
> to
> > > >> know that
> > > >> > > > the
> > > >> > > > > loggers are not supported in 1.5.
> > > >> > > > >
> > > >> > > > > *3.1.3
> > > >> > > > > Accumulo supports reading system config from xml local
> file
> > > and
> > > >> then
> > > >> > > > > zookeeper (whats set in zookeeper takes precedence). The
> nice
> > > >> thing
> > > >> > > > about
> > > >> > > > > setting something a config zookeeper, is that process
> restart
> > is
> > > >> not
> > > >> > > > > required.*
> > > >> > > > >
> > > >> > > > > Good to know this too. Does this only affect Zookeeper? I
> > > should
> > > >> > > > > distinguish between processes that may need restarts to
> load a
> > > >> > > > > configuration vs those that do not. Is the Zookeeper
> process
> > > the
> > > >> only
> > > >> > > > one
> > > >> > > > > like that?
> > > >> > > > >
> > > >> > > > >
> > > >> > > > Not sure about zookeeper, I think it requires a restart.
> > Accumulo
> > > >> can
> > > >> > > > optionally store a lot of its configuration in zookeeper.
> When
> > > >> Accumulo
> > > >> > > > configuration stored in zookeeper is changed, Accumulo
> processes
> > > >> may pick
> > > >> > > > up these config changes immediately w/o restart. Some
> > properties
> > > >> do
> > > >> > > > require a restart when changed in zookeeper. This is
> documented
> > > in
> > > >> > > > Accumulo config documentation.
> > > >> > > >
> > > >> > > >
> > > >> > > > >
> > > >> > > > > *For section 3.2, are you proposing adding features to the
> > > >> existing
> > > >> > > > Ambari
> > > >> > > > > GUI? If so, are the Accumulo specific?*
> > > >> > > > >
> > > >> > > > > I am proposing to add features to the existing Ambari GUI.
> > The
> > > >> idea
> > > >> > > here
> > > >> > > > > is that something like user access controls can be
> generalized
> > > so
> > > >> that
> > > >> > > > any
> > > >> > > > > Hadoop component being managed by Ambari is able to utilize
> > > them.
> > > >> I'm
> > > >> > > > > hoping that the Ambari PMC is OK with me doing this. I do
> not
> > > >> want to
> > > >> > > > > introduce any Accumulo-specific features that would make no
> > > sense
> > > >> for
> > > >> > > > other
> > > >> > > > > components.
> > > >> > > > > *
> > > >> > > > > It seems that Ambari relies on RPMs. Its possible you may
> > have
> > > to
> > > >> > > spend
> > > >> > > > > time creating Accumulo RPMs that are useful for Ambari. You
> > may
> > > >> want to
> > > >> > > > > think about allocating some time for this. Christopher and
> > John
> > > >> can
> > > >> > > give
> > > >> > > > > you more info about the current state of Accumulo RPMs.*
> > > >> > > > >
> > > >> > > > > This is a very good point. While building RPMs is not
> > > difficult,
> > > >> it
> > > >> > > does
> > > >> > > > > require time to setup the configuration and test the builds.
> > I
> > > >> will
> > > >> > > > > include time for this in the proposal.
> > > >> > > > >
> > > >> > > >
> > > >> > > > I suppose I am really thinking about are scripts for starting
> > and
> > > >> > > stopping
> > > >> > > > Accumulo. For an RPM to be useful it should provide good
> > > >> /etc/init.d
> > > >> > > > scripts for starting and stopping Accumulo processes. I am
> not
> > > >> sure
> > > >> > > what
> > > >> > > > the status of these scripts is in the various RPMs built by
> > > >> Accumulo.
> > > >> > > This
> > > >> > > > may not even be a concern, I am not sure how Ambari starts and
> > > stop
> > > >> > > > services. If you have not already, its something to consider
> > > when
> > > >> > > > scheduling.
> > > >> > > >
> > > >> > > >
> > > >> > > > >
> > > >> > > > > Thanks again, Keith.
> > > >> > > > >
> > > >> > > > > Andres
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Thu, Apr 25, 2013 at 10:54 AM, Keith Turner <
> > > keith@deenlo.com>
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > About the REST API, I meant to ask if this could be used
> > from
> > > >> the
> > > >> > > > command
> > > >> > > > > > line currently? Or is too cumbersome?
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Thu, Apr 25, 2013 at 10:09 AM, Andres Danter <
> > > >> adanter@gmail.com>
> > > >> > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Yup. Looks like that's the way to go. Here is the link
> > to
> > > >> the
> > > >> > > > > document.
> > > >> > > > > > > Let me know if you have any problems accessing it.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >>
> > >
> >
> https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Thu, Apr 25, 2013 at 9:51 AM, Keith Turner <
> > > >> keith@deenlo.com>
> > > >> > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Nothing. I suppose the mailing list is stripping it
> > off.
> > > >> Could
> > > >> > > > post
> > > >> > > > > > it
> > > >> > > > > > > as
> > > >> > > > > > > > an attachment to the ticket or put it on google drive
> or
> > > >> dropbox
> > > >> > > > and
> > > >> > > > > > > send a
> > > >> > > > > > > > link.
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Thu, Apr 25, 2013 at 9:44 AM, Andres Danter <
> > > >> > > adanter@gmail.com>
> > > >> > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Weird. I see the attachment on the email I sent,
> > which
> > > >> means
> > > >> > > > that
> > > >> > > > > it
> > > >> > > > > > > > > might have been stripped off. Here it is again.
> Let
> > me
> > > >> know
> > > >> > > if
> > > >> > > > > you
> > > >> > > > > > > > still
> > > >> > > > > > > > > don't see it.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Andres
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Thu, Apr 25, 2013 at 9:18 AM, Keith Turner <
> > > >> > > keith@deenlo.com>
> > > >> > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > >> was there supposed to be an attachment?
> > > >> > > > > > > > >>
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> On Thu, Apr 25, 2013 at 2:19 AM, Andres Danter <
> > > >> > > > adanter@gmail.com
> > > >> > > > > >
> > > >> > > > > > > > wrote:
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> > Hi Billie,
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Here is a draft of my proposal. I've yet to
> write
> > > the
> > > >> > > > technical
> > > >> > > > > > > > >> approach
> > > >> > > > > > > > >> > section (have not idea what I'm going to put
> there)
> > > >> and the
> > > >> > > > > "about
> > > >> > > > > > > me"
> > > >> > > > > > > > >> > section. I definitely appreciate any feedback
> from
> > > >> you and
> > > >> > > > > anyone
> > > >> > > > > > > in
> > > >> > > > > > > > >> the
> > > >> > > > > > > > >> > development group.
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Thank you,
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Andres
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >>
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >>
> > > >>
> > > >
> > > >
> > >
> >
>
Re: GSoC proposal: Ambari for Accumulo - Final Draft
Posted by Billie Rinaldi <bi...@gmail.com>.
I think including it in the proposal isn't necessary. I just wanted to
give you a heads up. :-)
Billie
On Wed, May 1, 2013 at 7:28 AM, Andres Danter <ad...@gmail.com> wrote:
> Thanks, Billie. I agree about working with 1.5.0. I was going to be
> sneaky and try to integrate with both versions, but I didn't want to put it
> on the proposal in case I run into complications. Let me know if you want
> me to include that in the proposal, though, and I'll add it.
>
> Thanks again.
>
> Andres
>
>
> On Wed, May 1, 2013 at 10:20 AM, Billie Rinaldi <billie.rinaldi@gmail.com
> >wrote:
>
> > Andres,
> >
> > The proposal looks good. I believe Accumulo will work with Hadoop 1.0.4.
> > You may want to work both with Accumulo 1.5.0 and 1.4.3 if we get 1.5.0
> > released soon enough.
> >
> > Billie
> >
> >
> > On Tue, Apr 30, 2013 at 10:25 PM, Andres Danter <ad...@gmail.com>
> wrote:
> >
> > > Hello Ambari and Accumulo developers,
> > >
> > > Here is the link to the final draft of my proposal. Not too much has
> > > changed. Some of the language on the previous draft was confusing, so
> I
> > > cleaned that up, and I did add some Ambari tidbits. The biggest
> > addition,
> > > in my opinion, is the table that lists the different versions of
> > software I
> > > will be using. Please take a look at that table to ensure I don't
> have a
> > > mismatch. I have a nagging feeling that the version of Hadoop that
> > Ambari
> > > can deploy is newer than the version Accumulo 1.4.3 is comfortable
> with.
> > >
> > > Thank you all again for your support and feedback. Here is the link:
> > >
> > >
> > >
> >
> https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
> > >
> > >
> > > Andres
> > >
> > >
> > > On Mon, Apr 29, 2013 at 8:03 PM, Andres Danter <ad...@gmail.com>
> > wrote:
> > >
> > > > Hi Drew,
> > > >
> > > > Sorry for replying so late. Unfortunately, this project is a solo
> > thing.
> > > > Hopefully we can team up on another project in the future.
> > > >
> > > > Thanks for the support.
> > > >
> > > > Andres
> > > >
> > > >
> > > > On Fri, Apr 26, 2013 at 11:01 AM, Drew Pierce <dr...@live.com>
> > > wrote:
> > > >
> > > >> Not sure if these are solo initiatives or those needing assistance.
> > Let
> > > >> me know if it is the latter :)
> > > >>
> > > >>
> > > >> > Date: Fri, 26 Apr 2013 10:47:21 -0400
> > > >> > Subject: Re: GSoC proposal: Ambari for Accumulo - forever : )
> > > >> > From: adanter@gmail.com
> > > >> > To: dev@accumulo.apache.org
> > > >> >
> > > >> > Thanks, Drew, for the enthusiam.
> > > >> >
> > > >> > And thanks, John, for the good information. It is definitely an
> > area
> > > I
> > > >> > need to look into.
> > > >> > On Apr 26, 2013 10:31 AM, "John Vines" <vi...@apache.org> wrote:
> > > >> >
> > > >> > > The init.d scripts we provide are well tested, provided you have
> > > >> accumulo
> > > >> > > installed into /usr/lib. There is probably room for improvement
> to
> > > >> utilize
> > > >> > > /etc/defaults/accumulo to make it configurable. This makes them
> > not
> > > >> > > configurable with the existing RPM packaging internal to
> Accumulo.
> > > And
> > > >> > > Ambari uses the init.d scripts to start/stop things.
> > > >> > >
> > > >> > > As mentioned in the other thread, I'm pretty sure Ambari is
> pretty
> > > >> tightly
> > > >> > > integrated with Bigtop. There is an Accumulo ticket for bigtop
> > > >> support (
> > > >> > > Accumulo-138 ). I went ahead and threw a patch on there in which
> > the
> > > >> RPM
> > > >> > > generated has been tested (the deb is lacking though).
> > Furthermore,
> > > >> they do
> > > >> > > not use the given Accumulo init.d scripts, so there may be room
> > for
> > > >> > > improvement in there somewhere, but I'm not sure.
> > > >> > >
> > > >> > > Should you need to work down that path more, I can definitely
> > > provide
> > > >> more
> > > >> > > guidance.
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Apr 25, 2013 at 12:18 PM, Keith Turner <
> keith@deenlo.com>
> > > >> wrote:
> > > >> > >
> > > >> > > > On Thu, Apr 25, 2013 at 11:40 AM, Andres Danter <
> > > adanter@gmail.com>
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > Excellent feedback, Keith. Thanks.
> > > >> > > > >
> > > >> > > > > I should have mentioned the REST API in the proposal. I'm
> not
> > > >> sure if
> > > >> > > I
> > > >> > > > > can use it for the CLI. I will certainly try, since I would
> > > like
> > > >> to
> > > >> > > > adhere
> > > >> > > > > to the Ambari architecture as much as possible.
> > > >> > > > >
> > > >> > > > > As for your other points:
> > > >> > > > >
> > > >> > > > > *3.1.1 loggers do not exists in 1.5 ... so would only need
> to
> > > >> consider
> > > >> > > > them
> > > >> > > > > if working w/ 1.4 . I suppose you would want to work
> Accumulo
> > > >> 1.4,
> > > >> > > since
> > > >> > > > > that the current stable release.*
> > > >> > > > >
> > > >> > > > > Yes. I was going to work with Accumulo 1.4, but it is good
> to
> > > >> know that
> > > >> > > > the
> > > >> > > > > loggers are not supported in 1.5.
> > > >> > > > >
> > > >> > > > > *3.1.3
> > > >> > > > > Accumulo supports reading system config from xml local
> file
> > > and
> > > >> then
> > > >> > > > > zookeeper (whats set in zookeeper takes precedence). The
> nice
> > > >> thing
> > > >> > > > about
> > > >> > > > > setting something a config zookeeper, is that process
> restart
> > is
> > > >> not
> > > >> > > > > required.*
> > > >> > > > >
> > > >> > > > > Good to know this too. Does this only affect Zookeeper? I
> > > should
> > > >> > > > > distinguish between processes that may need restarts to
> load a
> > > >> > > > > configuration vs those that do not. Is the Zookeeper
> process
> > > the
> > > >> only
> > > >> > > > one
> > > >> > > > > like that?
> > > >> > > > >
> > > >> > > > >
> > > >> > > > Not sure about zookeeper, I think it requires a restart.
> > Accumulo
> > > >> can
> > > >> > > > optionally store a lot of its configuration in zookeeper.
> When
> > > >> Accumulo
> > > >> > > > configuration stored in zookeeper is changed, Accumulo
> processes
> > > >> may pick
> > > >> > > > up these config changes immediately w/o restart. Some
> > properties
> > > >> do
> > > >> > > > require a restart when changed in zookeeper. This is
> documented
> > > in
> > > >> > > > Accumulo config documentation.
> > > >> > > >
> > > >> > > >
> > > >> > > > >
> > > >> > > > > *For section 3.2, are you proposing adding features to the
> > > >> existing
> > > >> > > > Ambari
> > > >> > > > > GUI? If so, are the Accumulo specific?*
> > > >> > > > >
> > > >> > > > > I am proposing to add features to the existing Ambari GUI.
> > The
> > > >> idea
> > > >> > > here
> > > >> > > > > is that something like user access controls can be
> generalized
> > > so
> > > >> that
> > > >> > > > any
> > > >> > > > > Hadoop component being managed by Ambari is able to utilize
> > > them.
> > > >> I'm
> > > >> > > > > hoping that the Ambari PMC is OK with me doing this. I do
> not
> > > >> want to
> > > >> > > > > introduce any Accumulo-specific features that would make no
> > > sense
> > > >> for
> > > >> > > > other
> > > >> > > > > components.
> > > >> > > > > *
> > > >> > > > > It seems that Ambari relies on RPMs. Its possible you may
> > have
> > > to
> > > >> > > spend
> > > >> > > > > time creating Accumulo RPMs that are useful for Ambari. You
> > may
> > > >> want to
> > > >> > > > > think about allocating some time for this. Christopher and
> > John
> > > >> can
> > > >> > > give
> > > >> > > > > you more info about the current state of Accumulo RPMs.*
> > > >> > > > >
> > > >> > > > > This is a very good point. While building RPMs is not
> > > difficult,
> > > >> it
> > > >> > > does
> > > >> > > > > require time to setup the configuration and test the builds.
> > I
> > > >> will
> > > >> > > > > include time for this in the proposal.
> > > >> > > > >
> > > >> > > >
> > > >> > > > I suppose I am really thinking about are scripts for starting
> > and
> > > >> > > stopping
> > > >> > > > Accumulo. For an RPM to be useful it should provide good
> > > >> /etc/init.d
> > > >> > > > scripts for starting and stopping Accumulo processes. I am
> not
> > > >> sure
> > > >> > > what
> > > >> > > > the status of these scripts is in the various RPMs built by
> > > >> Accumulo.
> > > >> > > This
> > > >> > > > may not even be a concern, I am not sure how Ambari starts and
> > > stop
> > > >> > > > services. If you have not already, its something to consider
> > > when
> > > >> > > > scheduling.
> > > >> > > >
> > > >> > > >
> > > >> > > > >
> > > >> > > > > Thanks again, Keith.
> > > >> > > > >
> > > >> > > > > Andres
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Thu, Apr 25, 2013 at 10:54 AM, Keith Turner <
> > > keith@deenlo.com>
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > About the REST API, I meant to ask if this could be used
> > from
> > > >> the
> > > >> > > > command
> > > >> > > > > > line currently? Or is too cumbersome?
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Thu, Apr 25, 2013 at 10:09 AM, Andres Danter <
> > > >> adanter@gmail.com>
> > > >> > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Yup. Looks like that's the way to go. Here is the link
> > to
> > > >> the
> > > >> > > > > document.
> > > >> > > > > > > Let me know if you have any problems accessing it.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >>
> > >
> >
> https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Thu, Apr 25, 2013 at 9:51 AM, Keith Turner <
> > > >> keith@deenlo.com>
> > > >> > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Nothing. I suppose the mailing list is stripping it
> > off.
> > > >> Could
> > > >> > > > post
> > > >> > > > > > it
> > > >> > > > > > > as
> > > >> > > > > > > > an attachment to the ticket or put it on google drive
> or
> > > >> dropbox
> > > >> > > > and
> > > >> > > > > > > send a
> > > >> > > > > > > > link.
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Thu, Apr 25, 2013 at 9:44 AM, Andres Danter <
> > > >> > > adanter@gmail.com>
> > > >> > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Weird. I see the attachment on the email I sent,
> > which
> > > >> means
> > > >> > > > that
> > > >> > > > > it
> > > >> > > > > > > > > might have been stripped off. Here it is again.
> Let
> > me
> > > >> know
> > > >> > > if
> > > >> > > > > you
> > > >> > > > > > > > still
> > > >> > > > > > > > > don't see it.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Andres
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Thu, Apr 25, 2013 at 9:18 AM, Keith Turner <
> > > >> > > keith@deenlo.com>
> > > >> > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > >> was there supposed to be an attachment?
> > > >> > > > > > > > >>
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> On Thu, Apr 25, 2013 at 2:19 AM, Andres Danter <
> > > >> > > > adanter@gmail.com
> > > >> > > > > >
> > > >> > > > > > > > wrote:
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> > Hi Billie,
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Here is a draft of my proposal. I've yet to
> write
> > > the
> > > >> > > > technical
> > > >> > > > > > > > >> approach
> > > >> > > > > > > > >> > section (have not idea what I'm going to put
> there)
> > > >> and the
> > > >> > > > > "about
> > > >> > > > > > > me"
> > > >> > > > > > > > >> > section. I definitely appreciate any feedback
> from
> > > >> you and
> > > >> > > > > anyone
> > > >> > > > > > > in
> > > >> > > > > > > > >> the
> > > >> > > > > > > > >> > development group.
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Thank you,
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Andres
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >>
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >>
> > > >>
> > > >
> > > >
> > >
> >
>
Re: GSoC proposal: Ambari for Accumulo - Final Draft
Posted by Andres Danter <ad...@gmail.com>.
Thanks, Billie. I agree about working with 1.5.0. I was going to be
sneaky and try to integrate with both versions, but I didn't want to put it
on the proposal in case I run into complications. Let me know if you want
me to include that in the proposal, though, and I'll add it.
Thanks again.
Andres
On Wed, May 1, 2013 at 10:20 AM, Billie Rinaldi <bi...@gmail.com>wrote:
> Andres,
>
> The proposal looks good. I believe Accumulo will work with Hadoop 1.0.4.
> You may want to work both with Accumulo 1.5.0 and 1.4.3 if we get 1.5.0
> released soon enough.
>
> Billie
>
>
> On Tue, Apr 30, 2013 at 10:25 PM, Andres Danter <ad...@gmail.com> wrote:
>
> > Hello Ambari and Accumulo developers,
> >
> > Here is the link to the final draft of my proposal. Not too much has
> > changed. Some of the language on the previous draft was confusing, so I
> > cleaned that up, and I did add some Ambari tidbits. The biggest
> addition,
> > in my opinion, is the table that lists the different versions of
> software I
> > will be using. Please take a look at that table to ensure I don't have a
> > mismatch. I have a nagging feeling that the version of Hadoop that
> Ambari
> > can deploy is newer than the version Accumulo 1.4.3 is comfortable with.
> >
> > Thank you all again for your support and feedback. Here is the link:
> >
> >
> >
> https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
> >
> >
> > Andres
> >
> >
> > On Mon, Apr 29, 2013 at 8:03 PM, Andres Danter <ad...@gmail.com>
> wrote:
> >
> > > Hi Drew,
> > >
> > > Sorry for replying so late. Unfortunately, this project is a solo
> thing.
> > > Hopefully we can team up on another project in the future.
> > >
> > > Thanks for the support.
> > >
> > > Andres
> > >
> > >
> > > On Fri, Apr 26, 2013 at 11:01 AM, Drew Pierce <dr...@live.com>
> > wrote:
> > >
> > >> Not sure if these are solo initiatives or those needing assistance.
> Let
> > >> me know if it is the latter :)
> > >>
> > >>
> > >> > Date: Fri, 26 Apr 2013 10:47:21 -0400
> > >> > Subject: Re: GSoC proposal: Ambari for Accumulo - forever : )
> > >> > From: adanter@gmail.com
> > >> > To: dev@accumulo.apache.org
> > >> >
> > >> > Thanks, Drew, for the enthusiam.
> > >> >
> > >> > And thanks, John, for the good information. It is definitely an
> area
> > I
> > >> > need to look into.
> > >> > On Apr 26, 2013 10:31 AM, "John Vines" <vi...@apache.org> wrote:
> > >> >
> > >> > > The init.d scripts we provide are well tested, provided you have
> > >> accumulo
> > >> > > installed into /usr/lib. There is probably room for improvement to
> > >> utilize
> > >> > > /etc/defaults/accumulo to make it configurable. This makes them
> not
> > >> > > configurable with the existing RPM packaging internal to Accumulo.
> > And
> > >> > > Ambari uses the init.d scripts to start/stop things.
> > >> > >
> > >> > > As mentioned in the other thread, I'm pretty sure Ambari is pretty
> > >> tightly
> > >> > > integrated with Bigtop. There is an Accumulo ticket for bigtop
> > >> support (
> > >> > > Accumulo-138 ). I went ahead and threw a patch on there in which
> the
> > >> RPM
> > >> > > generated has been tested (the deb is lacking though).
> Furthermore,
> > >> they do
> > >> > > not use the given Accumulo init.d scripts, so there may be room
> for
> > >> > > improvement in there somewhere, but I'm not sure.
> > >> > >
> > >> > > Should you need to work down that path more, I can definitely
> > provide
> > >> more
> > >> > > guidance.
> > >> > >
> > >> > >
> > >> > > On Thu, Apr 25, 2013 at 12:18 PM, Keith Turner <ke...@deenlo.com>
> > >> wrote:
> > >> > >
> > >> > > > On Thu, Apr 25, 2013 at 11:40 AM, Andres Danter <
> > adanter@gmail.com>
> > >> > > wrote:
> > >> > > >
> > >> > > > > Excellent feedback, Keith. Thanks.
> > >> > > > >
> > >> > > > > I should have mentioned the REST API in the proposal. I'm not
> > >> sure if
> > >> > > I
> > >> > > > > can use it for the CLI. I will certainly try, since I would
> > like
> > >> to
> > >> > > > adhere
> > >> > > > > to the Ambari architecture as much as possible.
> > >> > > > >
> > >> > > > > As for your other points:
> > >> > > > >
> > >> > > > > *3.1.1 loggers do not exists in 1.5 ... so would only need to
> > >> consider
> > >> > > > them
> > >> > > > > if working w/ 1.4 . I suppose you would want to work Accumulo
> > >> 1.4,
> > >> > > since
> > >> > > > > that the current stable release.*
> > >> > > > >
> > >> > > > > Yes. I was going to work with Accumulo 1.4, but it is good to
> > >> know that
> > >> > > > the
> > >> > > > > loggers are not supported in 1.5.
> > >> > > > >
> > >> > > > > *3.1.3
> > >> > > > > Accumulo supports reading system config from xml local file
> > and
> > >> then
> > >> > > > > zookeeper (whats set in zookeeper takes precedence). The nice
> > >> thing
> > >> > > > about
> > >> > > > > setting something a config zookeeper, is that process restart
> is
> > >> not
> > >> > > > > required.*
> > >> > > > >
> > >> > > > > Good to know this too. Does this only affect Zookeeper? I
> > should
> > >> > > > > distinguish between processes that may need restarts to load a
> > >> > > > > configuration vs those that do not. Is the Zookeeper process
> > the
> > >> only
> > >> > > > one
> > >> > > > > like that?
> > >> > > > >
> > >> > > > >
> > >> > > > Not sure about zookeeper, I think it requires a restart.
> Accumulo
> > >> can
> > >> > > > optionally store a lot of its configuration in zookeeper. When
> > >> Accumulo
> > >> > > > configuration stored in zookeeper is changed, Accumulo processes
> > >> may pick
> > >> > > > up these config changes immediately w/o restart. Some
> properties
> > >> do
> > >> > > > require a restart when changed in zookeeper. This is documented
> > in
> > >> > > > Accumulo config documentation.
> > >> > > >
> > >> > > >
> > >> > > > >
> > >> > > > > *For section 3.2, are you proposing adding features to the
> > >> existing
> > >> > > > Ambari
> > >> > > > > GUI? If so, are the Accumulo specific?*
> > >> > > > >
> > >> > > > > I am proposing to add features to the existing Ambari GUI.
> The
> > >> idea
> > >> > > here
> > >> > > > > is that something like user access controls can be generalized
> > so
> > >> that
> > >> > > > any
> > >> > > > > Hadoop component being managed by Ambari is able to utilize
> > them.
> > >> I'm
> > >> > > > > hoping that the Ambari PMC is OK with me doing this. I do not
> > >> want to
> > >> > > > > introduce any Accumulo-specific features that would make no
> > sense
> > >> for
> > >> > > > other
> > >> > > > > components.
> > >> > > > > *
> > >> > > > > It seems that Ambari relies on RPMs. Its possible you may
> have
> > to
> > >> > > spend
> > >> > > > > time creating Accumulo RPMs that are useful for Ambari. You
> may
> > >> want to
> > >> > > > > think about allocating some time for this. Christopher and
> John
> > >> can
> > >> > > give
> > >> > > > > you more info about the current state of Accumulo RPMs.*
> > >> > > > >
> > >> > > > > This is a very good point. While building RPMs is not
> > difficult,
> > >> it
> > >> > > does
> > >> > > > > require time to setup the configuration and test the builds.
> I
> > >> will
> > >> > > > > include time for this in the proposal.
> > >> > > > >
> > >> > > >
> > >> > > > I suppose I am really thinking about are scripts for starting
> and
> > >> > > stopping
> > >> > > > Accumulo. For an RPM to be useful it should provide good
> > >> /etc/init.d
> > >> > > > scripts for starting and stopping Accumulo processes. I am not
> > >> sure
> > >> > > what
> > >> > > > the status of these scripts is in the various RPMs built by
> > >> Accumulo.
> > >> > > This
> > >> > > > may not even be a concern, I am not sure how Ambari starts and
> > stop
> > >> > > > services. If you have not already, its something to consider
> > when
> > >> > > > scheduling.
> > >> > > >
> > >> > > >
> > >> > > > >
> > >> > > > > Thanks again, Keith.
> > >> > > > >
> > >> > > > > Andres
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > On Thu, Apr 25, 2013 at 10:54 AM, Keith Turner <
> > keith@deenlo.com>
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > About the REST API, I meant to ask if this could be used
> from
> > >> the
> > >> > > > command
> > >> > > > > > line currently? Or is too cumbersome?
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Thu, Apr 25, 2013 at 10:09 AM, Andres Danter <
> > >> adanter@gmail.com>
> > >> > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Yup. Looks like that's the way to go. Here is the link
> to
> > >> the
> > >> > > > > document.
> > >> > > > > > > Let me know if you have any problems accessing it.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >>
> >
> https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Thu, Apr 25, 2013 at 9:51 AM, Keith Turner <
> > >> keith@deenlo.com>
> > >> > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Nothing. I suppose the mailing list is stripping it
> off.
> > >> Could
> > >> > > > post
> > >> > > > > > it
> > >> > > > > > > as
> > >> > > > > > > > an attachment to the ticket or put it on google drive or
> > >> dropbox
> > >> > > > and
> > >> > > > > > > send a
> > >> > > > > > > > link.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Thu, Apr 25, 2013 at 9:44 AM, Andres Danter <
> > >> > > adanter@gmail.com>
> > >> > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Weird. I see the attachment on the email I sent,
> which
> > >> means
> > >> > > > that
> > >> > > > > it
> > >> > > > > > > > > might have been stripped off. Here it is again. Let
> me
> > >> know
> > >> > > if
> > >> > > > > you
> > >> > > > > > > > still
> > >> > > > > > > > > don't see it.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > >
> > >> > > > > > > > > Andres
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Thu, Apr 25, 2013 at 9:18 AM, Keith Turner <
> > >> > > keith@deenlo.com>
> > >> > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > >> was there supposed to be an attachment?
> > >> > > > > > > > >>
> > >> > > > > > > > >>
> > >> > > > > > > > >> On Thu, Apr 25, 2013 at 2:19 AM, Andres Danter <
> > >> > > > adanter@gmail.com
> > >> > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > > >>
> > >> > > > > > > > >> > Hi Billie,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Here is a draft of my proposal. I've yet to write
> > the
> > >> > > > technical
> > >> > > > > > > > >> approach
> > >> > > > > > > > >> > section (have not idea what I'm going to put there)
> > >> and the
> > >> > > > > "about
> > >> > > > > > > me"
> > >> > > > > > > > >> > section. I definitely appreciate any feedback from
> > >> you and
> > >> > > > > anyone
> > >> > > > > > > in
> > >> > > > > > > > >> the
> > >> > > > > > > > >> > development group.
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Thank you,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Andres
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >>
> > >>
> > >
> > >
> >
>
Re: GSoC proposal: Ambari for Accumulo - Final Draft
Posted by Andres Danter <ad...@gmail.com>.
Thanks, Billie. I agree about working with 1.5.0. I was going to be
sneaky and try to integrate with both versions, but I didn't want to put it
on the proposal in case I run into complications. Let me know if you want
me to include that in the proposal, though, and I'll add it.
Thanks again.
Andres
On Wed, May 1, 2013 at 10:20 AM, Billie Rinaldi <bi...@gmail.com>wrote:
> Andres,
>
> The proposal looks good. I believe Accumulo will work with Hadoop 1.0.4.
> You may want to work both with Accumulo 1.5.0 and 1.4.3 if we get 1.5.0
> released soon enough.
>
> Billie
>
>
> On Tue, Apr 30, 2013 at 10:25 PM, Andres Danter <ad...@gmail.com> wrote:
>
> > Hello Ambari and Accumulo developers,
> >
> > Here is the link to the final draft of my proposal. Not too much has
> > changed. Some of the language on the previous draft was confusing, so I
> > cleaned that up, and I did add some Ambari tidbits. The biggest
> addition,
> > in my opinion, is the table that lists the different versions of
> software I
> > will be using. Please take a look at that table to ensure I don't have a
> > mismatch. I have a nagging feeling that the version of Hadoop that
> Ambari
> > can deploy is newer than the version Accumulo 1.4.3 is comfortable with.
> >
> > Thank you all again for your support and feedback. Here is the link:
> >
> >
> >
> https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
> >
> >
> > Andres
> >
> >
> > On Mon, Apr 29, 2013 at 8:03 PM, Andres Danter <ad...@gmail.com>
> wrote:
> >
> > > Hi Drew,
> > >
> > > Sorry for replying so late. Unfortunately, this project is a solo
> thing.
> > > Hopefully we can team up on another project in the future.
> > >
> > > Thanks for the support.
> > >
> > > Andres
> > >
> > >
> > > On Fri, Apr 26, 2013 at 11:01 AM, Drew Pierce <dr...@live.com>
> > wrote:
> > >
> > >> Not sure if these are solo initiatives or those needing assistance.
> Let
> > >> me know if it is the latter :)
> > >>
> > >>
> > >> > Date: Fri, 26 Apr 2013 10:47:21 -0400
> > >> > Subject: Re: GSoC proposal: Ambari for Accumulo - forever : )
> > >> > From: adanter@gmail.com
> > >> > To: dev@accumulo.apache.org
> > >> >
> > >> > Thanks, Drew, for the enthusiam.
> > >> >
> > >> > And thanks, John, for the good information. It is definitely an
> area
> > I
> > >> > need to look into.
> > >> > On Apr 26, 2013 10:31 AM, "John Vines" <vi...@apache.org> wrote:
> > >> >
> > >> > > The init.d scripts we provide are well tested, provided you have
> > >> accumulo
> > >> > > installed into /usr/lib. There is probably room for improvement to
> > >> utilize
> > >> > > /etc/defaults/accumulo to make it configurable. This makes them
> not
> > >> > > configurable with the existing RPM packaging internal to Accumulo.
> > And
> > >> > > Ambari uses the init.d scripts to start/stop things.
> > >> > >
> > >> > > As mentioned in the other thread, I'm pretty sure Ambari is pretty
> > >> tightly
> > >> > > integrated with Bigtop. There is an Accumulo ticket for bigtop
> > >> support (
> > >> > > Accumulo-138 ). I went ahead and threw a patch on there in which
> the
> > >> RPM
> > >> > > generated has been tested (the deb is lacking though).
> Furthermore,
> > >> they do
> > >> > > not use the given Accumulo init.d scripts, so there may be room
> for
> > >> > > improvement in there somewhere, but I'm not sure.
> > >> > >
> > >> > > Should you need to work down that path more, I can definitely
> > provide
> > >> more
> > >> > > guidance.
> > >> > >
> > >> > >
> > >> > > On Thu, Apr 25, 2013 at 12:18 PM, Keith Turner <ke...@deenlo.com>
> > >> wrote:
> > >> > >
> > >> > > > On Thu, Apr 25, 2013 at 11:40 AM, Andres Danter <
> > adanter@gmail.com>
> > >> > > wrote:
> > >> > > >
> > >> > > > > Excellent feedback, Keith. Thanks.
> > >> > > > >
> > >> > > > > I should have mentioned the REST API in the proposal. I'm not
> > >> sure if
> > >> > > I
> > >> > > > > can use it for the CLI. I will certainly try, since I would
> > like
> > >> to
> > >> > > > adhere
> > >> > > > > to the Ambari architecture as much as possible.
> > >> > > > >
> > >> > > > > As for your other points:
> > >> > > > >
> > >> > > > > *3.1.1 loggers do not exists in 1.5 ... so would only need to
> > >> consider
> > >> > > > them
> > >> > > > > if working w/ 1.4 . I suppose you would want to work Accumulo
> > >> 1.4,
> > >> > > since
> > >> > > > > that the current stable release.*
> > >> > > > >
> > >> > > > > Yes. I was going to work with Accumulo 1.4, but it is good to
> > >> know that
> > >> > > > the
> > >> > > > > loggers are not supported in 1.5.
> > >> > > > >
> > >> > > > > *3.1.3
> > >> > > > > Accumulo supports reading system config from xml local file
> > and
> > >> then
> > >> > > > > zookeeper (whats set in zookeeper takes precedence). The nice
> > >> thing
> > >> > > > about
> > >> > > > > setting something a config zookeeper, is that process restart
> is
> > >> not
> > >> > > > > required.*
> > >> > > > >
> > >> > > > > Good to know this too. Does this only affect Zookeeper? I
> > should
> > >> > > > > distinguish between processes that may need restarts to load a
> > >> > > > > configuration vs those that do not. Is the Zookeeper process
> > the
> > >> only
> > >> > > > one
> > >> > > > > like that?
> > >> > > > >
> > >> > > > >
> > >> > > > Not sure about zookeeper, I think it requires a restart.
> Accumulo
> > >> can
> > >> > > > optionally store a lot of its configuration in zookeeper. When
> > >> Accumulo
> > >> > > > configuration stored in zookeeper is changed, Accumulo processes
> > >> may pick
> > >> > > > up these config changes immediately w/o restart. Some
> properties
> > >> do
> > >> > > > require a restart when changed in zookeeper. This is documented
> > in
> > >> > > > Accumulo config documentation.
> > >> > > >
> > >> > > >
> > >> > > > >
> > >> > > > > *For section 3.2, are you proposing adding features to the
> > >> existing
> > >> > > > Ambari
> > >> > > > > GUI? If so, are the Accumulo specific?*
> > >> > > > >
> > >> > > > > I am proposing to add features to the existing Ambari GUI.
> The
> > >> idea
> > >> > > here
> > >> > > > > is that something like user access controls can be generalized
> > so
> > >> that
> > >> > > > any
> > >> > > > > Hadoop component being managed by Ambari is able to utilize
> > them.
> > >> I'm
> > >> > > > > hoping that the Ambari PMC is OK with me doing this. I do not
> > >> want to
> > >> > > > > introduce any Accumulo-specific features that would make no
> > sense
> > >> for
> > >> > > > other
> > >> > > > > components.
> > >> > > > > *
> > >> > > > > It seems that Ambari relies on RPMs. Its possible you may
> have
> > to
> > >> > > spend
> > >> > > > > time creating Accumulo RPMs that are useful for Ambari. You
> may
> > >> want to
> > >> > > > > think about allocating some time for this. Christopher and
> John
> > >> can
> > >> > > give
> > >> > > > > you more info about the current state of Accumulo RPMs.*
> > >> > > > >
> > >> > > > > This is a very good point. While building RPMs is not
> > difficult,
> > >> it
> > >> > > does
> > >> > > > > require time to setup the configuration and test the builds.
> I
> > >> will
> > >> > > > > include time for this in the proposal.
> > >> > > > >
> > >> > > >
> > >> > > > I suppose I am really thinking about are scripts for starting
> and
> > >> > > stopping
> > >> > > > Accumulo. For an RPM to be useful it should provide good
> > >> /etc/init.d
> > >> > > > scripts for starting and stopping Accumulo processes. I am not
> > >> sure
> > >> > > what
> > >> > > > the status of these scripts is in the various RPMs built by
> > >> Accumulo.
> > >> > > This
> > >> > > > may not even be a concern, I am not sure how Ambari starts and
> > stop
> > >> > > > services. If you have not already, its something to consider
> > when
> > >> > > > scheduling.
> > >> > > >
> > >> > > >
> > >> > > > >
> > >> > > > > Thanks again, Keith.
> > >> > > > >
> > >> > > > > Andres
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > On Thu, Apr 25, 2013 at 10:54 AM, Keith Turner <
> > keith@deenlo.com>
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > About the REST API, I meant to ask if this could be used
> from
> > >> the
> > >> > > > command
> > >> > > > > > line currently? Or is too cumbersome?
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Thu, Apr 25, 2013 at 10:09 AM, Andres Danter <
> > >> adanter@gmail.com>
> > >> > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Yup. Looks like that's the way to go. Here is the link
> to
> > >> the
> > >> > > > > document.
> > >> > > > > > > Let me know if you have any problems accessing it.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >>
> >
> https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Thu, Apr 25, 2013 at 9:51 AM, Keith Turner <
> > >> keith@deenlo.com>
> > >> > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Nothing. I suppose the mailing list is stripping it
> off.
> > >> Could
> > >> > > > post
> > >> > > > > > it
> > >> > > > > > > as
> > >> > > > > > > > an attachment to the ticket or put it on google drive or
> > >> dropbox
> > >> > > > and
> > >> > > > > > > send a
> > >> > > > > > > > link.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Thu, Apr 25, 2013 at 9:44 AM, Andres Danter <
> > >> > > adanter@gmail.com>
> > >> > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Weird. I see the attachment on the email I sent,
> which
> > >> means
> > >> > > > that
> > >> > > > > it
> > >> > > > > > > > > might have been stripped off. Here it is again. Let
> me
> > >> know
> > >> > > if
> > >> > > > > you
> > >> > > > > > > > still
> > >> > > > > > > > > don't see it.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > >
> > >> > > > > > > > > Andres
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Thu, Apr 25, 2013 at 9:18 AM, Keith Turner <
> > >> > > keith@deenlo.com>
> > >> > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > >> was there supposed to be an attachment?
> > >> > > > > > > > >>
> > >> > > > > > > > >>
> > >> > > > > > > > >> On Thu, Apr 25, 2013 at 2:19 AM, Andres Danter <
> > >> > > > adanter@gmail.com
> > >> > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > > >>
> > >> > > > > > > > >> > Hi Billie,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Here is a draft of my proposal. I've yet to write
> > the
> > >> > > > technical
> > >> > > > > > > > >> approach
> > >> > > > > > > > >> > section (have not idea what I'm going to put there)
> > >> and the
> > >> > > > > "about
> > >> > > > > > > me"
> > >> > > > > > > > >> > section. I definitely appreciate any feedback from
> > >> you and
> > >> > > > > anyone
> > >> > > > > > > in
> > >> > > > > > > > >> the
> > >> > > > > > > > >> > development group.
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Thank you,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Andres
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >>
> > >>
> > >
> > >
> >
>
Re: GSoC proposal: Ambari for Accumulo - Final Draft
Posted by Billie Rinaldi <bi...@gmail.com>.
Andres,
The proposal looks good. I believe Accumulo will work with Hadoop 1.0.4.
You may want to work both with Accumulo 1.5.0 and 1.4.3 if we get 1.5.0
released soon enough.
Billie
On Tue, Apr 30, 2013 at 10:25 PM, Andres Danter <ad...@gmail.com> wrote:
> Hello Ambari and Accumulo developers,
>
> Here is the link to the final draft of my proposal. Not too much has
> changed. Some of the language on the previous draft was confusing, so I
> cleaned that up, and I did add some Ambari tidbits. The biggest addition,
> in my opinion, is the table that lists the different versions of software I
> will be using. Please take a look at that table to ensure I don't have a
> mismatch. I have a nagging feeling that the version of Hadoop that Ambari
> can deploy is newer than the version Accumulo 1.4.3 is comfortable with.
>
> Thank you all again for your support and feedback. Here is the link:
>
>
> https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
>
>
> Andres
>
>
> On Mon, Apr 29, 2013 at 8:03 PM, Andres Danter <ad...@gmail.com> wrote:
>
> > Hi Drew,
> >
> > Sorry for replying so late. Unfortunately, this project is a solo thing.
> > Hopefully we can team up on another project in the future.
> >
> > Thanks for the support.
> >
> > Andres
> >
> >
> > On Fri, Apr 26, 2013 at 11:01 AM, Drew Pierce <dr...@live.com>
> wrote:
> >
> >> Not sure if these are solo initiatives or those needing assistance. Let
> >> me know if it is the latter :)
> >>
> >>
> >> > Date: Fri, 26 Apr 2013 10:47:21 -0400
> >> > Subject: Re: GSoC proposal: Ambari for Accumulo - forever : )
> >> > From: adanter@gmail.com
> >> > To: dev@accumulo.apache.org
> >> >
> >> > Thanks, Drew, for the enthusiam.
> >> >
> >> > And thanks, John, for the good information. It is definitely an area
> I
> >> > need to look into.
> >> > On Apr 26, 2013 10:31 AM, "John Vines" <vi...@apache.org> wrote:
> >> >
> >> > > The init.d scripts we provide are well tested, provided you have
> >> accumulo
> >> > > installed into /usr/lib. There is probably room for improvement to
> >> utilize
> >> > > /etc/defaults/accumulo to make it configurable. This makes them not
> >> > > configurable with the existing RPM packaging internal to Accumulo.
> And
> >> > > Ambari uses the init.d scripts to start/stop things.
> >> > >
> >> > > As mentioned in the other thread, I'm pretty sure Ambari is pretty
> >> tightly
> >> > > integrated with Bigtop. There is an Accumulo ticket for bigtop
> >> support (
> >> > > Accumulo-138 ). I went ahead and threw a patch on there in which the
> >> RPM
> >> > > generated has been tested (the deb is lacking though). Furthermore,
> >> they do
> >> > > not use the given Accumulo init.d scripts, so there may be room for
> >> > > improvement in there somewhere, but I'm not sure.
> >> > >
> >> > > Should you need to work down that path more, I can definitely
> provide
> >> more
> >> > > guidance.
> >> > >
> >> > >
> >> > > On Thu, Apr 25, 2013 at 12:18 PM, Keith Turner <ke...@deenlo.com>
> >> wrote:
> >> > >
> >> > > > On Thu, Apr 25, 2013 at 11:40 AM, Andres Danter <
> adanter@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > > Excellent feedback, Keith. Thanks.
> >> > > > >
> >> > > > > I should have mentioned the REST API in the proposal. I'm not
> >> sure if
> >> > > I
> >> > > > > can use it for the CLI. I will certainly try, since I would
> like
> >> to
> >> > > > adhere
> >> > > > > to the Ambari architecture as much as possible.
> >> > > > >
> >> > > > > As for your other points:
> >> > > > >
> >> > > > > *3.1.1 loggers do not exists in 1.5 ... so would only need to
> >> consider
> >> > > > them
> >> > > > > if working w/ 1.4 . I suppose you would want to work Accumulo
> >> 1.4,
> >> > > since
> >> > > > > that the current stable release.*
> >> > > > >
> >> > > > > Yes. I was going to work with Accumulo 1.4, but it is good to
> >> know that
> >> > > > the
> >> > > > > loggers are not supported in 1.5.
> >> > > > >
> >> > > > > *3.1.3
> >> > > > > Accumulo supports reading system config from xml local file
> and
> >> then
> >> > > > > zookeeper (whats set in zookeeper takes precedence). The nice
> >> thing
> >> > > > about
> >> > > > > setting something a config zookeeper, is that process restart is
> >> not
> >> > > > > required.*
> >> > > > >
> >> > > > > Good to know this too. Does this only affect Zookeeper? I
> should
> >> > > > > distinguish between processes that may need restarts to load a
> >> > > > > configuration vs those that do not. Is the Zookeeper process
> the
> >> only
> >> > > > one
> >> > > > > like that?
> >> > > > >
> >> > > > >
> >> > > > Not sure about zookeeper, I think it requires a restart. Accumulo
> >> can
> >> > > > optionally store a lot of its configuration in zookeeper. When
> >> Accumulo
> >> > > > configuration stored in zookeeper is changed, Accumulo processes
> >> may pick
> >> > > > up these config changes immediately w/o restart. Some properties
> >> do
> >> > > > require a restart when changed in zookeeper. This is documented
> in
> >> > > > Accumulo config documentation.
> >> > > >
> >> > > >
> >> > > > >
> >> > > > > *For section 3.2, are you proposing adding features to the
> >> existing
> >> > > > Ambari
> >> > > > > GUI? If so, are the Accumulo specific?*
> >> > > > >
> >> > > > > I am proposing to add features to the existing Ambari GUI. The
> >> idea
> >> > > here
> >> > > > > is that something like user access controls can be generalized
> so
> >> that
> >> > > > any
> >> > > > > Hadoop component being managed by Ambari is able to utilize
> them.
> >> I'm
> >> > > > > hoping that the Ambari PMC is OK with me doing this. I do not
> >> want to
> >> > > > > introduce any Accumulo-specific features that would make no
> sense
> >> for
> >> > > > other
> >> > > > > components.
> >> > > > > *
> >> > > > > It seems that Ambari relies on RPMs. Its possible you may have
> to
> >> > > spend
> >> > > > > time creating Accumulo RPMs that are useful for Ambari. You may
> >> want to
> >> > > > > think about allocating some time for this. Christopher and John
> >> can
> >> > > give
> >> > > > > you more info about the current state of Accumulo RPMs.*
> >> > > > >
> >> > > > > This is a very good point. While building RPMs is not
> difficult,
> >> it
> >> > > does
> >> > > > > require time to setup the configuration and test the builds. I
> >> will
> >> > > > > include time for this in the proposal.
> >> > > > >
> >> > > >
> >> > > > I suppose I am really thinking about are scripts for starting and
> >> > > stopping
> >> > > > Accumulo. For an RPM to be useful it should provide good
> >> /etc/init.d
> >> > > > scripts for starting and stopping Accumulo processes. I am not
> >> sure
> >> > > what
> >> > > > the status of these scripts is in the various RPMs built by
> >> Accumulo.
> >> > > This
> >> > > > may not even be a concern, I am not sure how Ambari starts and
> stop
> >> > > > services. If you have not already, its something to consider
> when
> >> > > > scheduling.
> >> > > >
> >> > > >
> >> > > > >
> >> > > > > Thanks again, Keith.
> >> > > > >
> >> > > > > Andres
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Thu, Apr 25, 2013 at 10:54 AM, Keith Turner <
> keith@deenlo.com>
> >> > > wrote:
> >> > > > >
> >> > > > > > About the REST API, I meant to ask if this could be used from
> >> the
> >> > > > command
> >> > > > > > line currently? Or is too cumbersome?
> >> > > > > >
> >> > > > > >
> >> > > > > > On Thu, Apr 25, 2013 at 10:09 AM, Andres Danter <
> >> adanter@gmail.com>
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > > Yup. Looks like that's the way to go. Here is the link to
> >> the
> >> > > > > document.
> >> > > > > > > Let me know if you have any problems accessing it.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >>
> https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Thu, Apr 25, 2013 at 9:51 AM, Keith Turner <
> >> keith@deenlo.com>
> >> > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Nothing. I suppose the mailing list is stripping it off.
> >> Could
> >> > > > post
> >> > > > > > it
> >> > > > > > > as
> >> > > > > > > > an attachment to the ticket or put it on google drive or
> >> dropbox
> >> > > > and
> >> > > > > > > send a
> >> > > > > > > > link.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Thu, Apr 25, 2013 at 9:44 AM, Andres Danter <
> >> > > adanter@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Weird. I see the attachment on the email I sent, which
> >> means
> >> > > > that
> >> > > > > it
> >> > > > > > > > > might have been stripped off. Here it is again. Let me
> >> know
> >> > > if
> >> > > > > you
> >> > > > > > > > still
> >> > > > > > > > > don't see it.
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > >
> >> > > > > > > > > Andres
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Thu, Apr 25, 2013 at 9:18 AM, Keith Turner <
> >> > > keith@deenlo.com>
> >> > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > >> was there supposed to be an attachment?
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > > > >> On Thu, Apr 25, 2013 at 2:19 AM, Andres Danter <
> >> > > > adanter@gmail.com
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > > >>
> >> > > > > > > > >> > Hi Billie,
> >> > > > > > > > >> >
> >> > > > > > > > >> > Here is a draft of my proposal. I've yet to write
> the
> >> > > > technical
> >> > > > > > > > >> approach
> >> > > > > > > > >> > section (have not idea what I'm going to put there)
> >> and the
> >> > > > > "about
> >> > > > > > > me"
> >> > > > > > > > >> > section. I definitely appreciate any feedback from
> >> you and
> >> > > > > anyone
> >> > > > > > > in
> >> > > > > > > > >> the
> >> > > > > > > > >> > development group.
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thank you,
> >> > > > > > > > >> >
> >> > > > > > > > >> > Andres
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >>
> >>
> >
> >
>
Re: GSoC proposal: Ambari for Accumulo - Final Draft
Posted by Billie Rinaldi <bi...@gmail.com>.
Andres,
The proposal looks good. I believe Accumulo will work with Hadoop 1.0.4.
You may want to work both with Accumulo 1.5.0 and 1.4.3 if we get 1.5.0
released soon enough.
Billie
On Tue, Apr 30, 2013 at 10:25 PM, Andres Danter <ad...@gmail.com> wrote:
> Hello Ambari and Accumulo developers,
>
> Here is the link to the final draft of my proposal. Not too much has
> changed. Some of the language on the previous draft was confusing, so I
> cleaned that up, and I did add some Ambari tidbits. The biggest addition,
> in my opinion, is the table that lists the different versions of software I
> will be using. Please take a look at that table to ensure I don't have a
> mismatch. I have a nagging feeling that the version of Hadoop that Ambari
> can deploy is newer than the version Accumulo 1.4.3 is comfortable with.
>
> Thank you all again for your support and feedback. Here is the link:
>
>
> https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
>
>
> Andres
>
>
> On Mon, Apr 29, 2013 at 8:03 PM, Andres Danter <ad...@gmail.com> wrote:
>
> > Hi Drew,
> >
> > Sorry for replying so late. Unfortunately, this project is a solo thing.
> > Hopefully we can team up on another project in the future.
> >
> > Thanks for the support.
> >
> > Andres
> >
> >
> > On Fri, Apr 26, 2013 at 11:01 AM, Drew Pierce <dr...@live.com>
> wrote:
> >
> >> Not sure if these are solo initiatives or those needing assistance. Let
> >> me know if it is the latter :)
> >>
> >>
> >> > Date: Fri, 26 Apr 2013 10:47:21 -0400
> >> > Subject: Re: GSoC proposal: Ambari for Accumulo - forever : )
> >> > From: adanter@gmail.com
> >> > To: dev@accumulo.apache.org
> >> >
> >> > Thanks, Drew, for the enthusiam.
> >> >
> >> > And thanks, John, for the good information. It is definitely an area
> I
> >> > need to look into.
> >> > On Apr 26, 2013 10:31 AM, "John Vines" <vi...@apache.org> wrote:
> >> >
> >> > > The init.d scripts we provide are well tested, provided you have
> >> accumulo
> >> > > installed into /usr/lib. There is probably room for improvement to
> >> utilize
> >> > > /etc/defaults/accumulo to make it configurable. This makes them not
> >> > > configurable with the existing RPM packaging internal to Accumulo.
> And
> >> > > Ambari uses the init.d scripts to start/stop things.
> >> > >
> >> > > As mentioned in the other thread, I'm pretty sure Ambari is pretty
> >> tightly
> >> > > integrated with Bigtop. There is an Accumulo ticket for bigtop
> >> support (
> >> > > Accumulo-138 ). I went ahead and threw a patch on there in which the
> >> RPM
> >> > > generated has been tested (the deb is lacking though). Furthermore,
> >> they do
> >> > > not use the given Accumulo init.d scripts, so there may be room for
> >> > > improvement in there somewhere, but I'm not sure.
> >> > >
> >> > > Should you need to work down that path more, I can definitely
> provide
> >> more
> >> > > guidance.
> >> > >
> >> > >
> >> > > On Thu, Apr 25, 2013 at 12:18 PM, Keith Turner <ke...@deenlo.com>
> >> wrote:
> >> > >
> >> > > > On Thu, Apr 25, 2013 at 11:40 AM, Andres Danter <
> adanter@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > > Excellent feedback, Keith. Thanks.
> >> > > > >
> >> > > > > I should have mentioned the REST API in the proposal. I'm not
> >> sure if
> >> > > I
> >> > > > > can use it for the CLI. I will certainly try, since I would
> like
> >> to
> >> > > > adhere
> >> > > > > to the Ambari architecture as much as possible.
> >> > > > >
> >> > > > > As for your other points:
> >> > > > >
> >> > > > > *3.1.1 loggers do not exists in 1.5 ... so would only need to
> >> consider
> >> > > > them
> >> > > > > if working w/ 1.4 . I suppose you would want to work Accumulo
> >> 1.4,
> >> > > since
> >> > > > > that the current stable release.*
> >> > > > >
> >> > > > > Yes. I was going to work with Accumulo 1.4, but it is good to
> >> know that
> >> > > > the
> >> > > > > loggers are not supported in 1.5.
> >> > > > >
> >> > > > > *3.1.3
> >> > > > > Accumulo supports reading system config from xml local file
> and
> >> then
> >> > > > > zookeeper (whats set in zookeeper takes precedence). The nice
> >> thing
> >> > > > about
> >> > > > > setting something a config zookeeper, is that process restart is
> >> not
> >> > > > > required.*
> >> > > > >
> >> > > > > Good to know this too. Does this only affect Zookeeper? I
> should
> >> > > > > distinguish between processes that may need restarts to load a
> >> > > > > configuration vs those that do not. Is the Zookeeper process
> the
> >> only
> >> > > > one
> >> > > > > like that?
> >> > > > >
> >> > > > >
> >> > > > Not sure about zookeeper, I think it requires a restart. Accumulo
> >> can
> >> > > > optionally store a lot of its configuration in zookeeper. When
> >> Accumulo
> >> > > > configuration stored in zookeeper is changed, Accumulo processes
> >> may pick
> >> > > > up these config changes immediately w/o restart. Some properties
> >> do
> >> > > > require a restart when changed in zookeeper. This is documented
> in
> >> > > > Accumulo config documentation.
> >> > > >
> >> > > >
> >> > > > >
> >> > > > > *For section 3.2, are you proposing adding features to the
> >> existing
> >> > > > Ambari
> >> > > > > GUI? If so, are the Accumulo specific?*
> >> > > > >
> >> > > > > I am proposing to add features to the existing Ambari GUI. The
> >> idea
> >> > > here
> >> > > > > is that something like user access controls can be generalized
> so
> >> that
> >> > > > any
> >> > > > > Hadoop component being managed by Ambari is able to utilize
> them.
> >> I'm
> >> > > > > hoping that the Ambari PMC is OK with me doing this. I do not
> >> want to
> >> > > > > introduce any Accumulo-specific features that would make no
> sense
> >> for
> >> > > > other
> >> > > > > components.
> >> > > > > *
> >> > > > > It seems that Ambari relies on RPMs. Its possible you may have
> to
> >> > > spend
> >> > > > > time creating Accumulo RPMs that are useful for Ambari. You may
> >> want to
> >> > > > > think about allocating some time for this. Christopher and John
> >> can
> >> > > give
> >> > > > > you more info about the current state of Accumulo RPMs.*
> >> > > > >
> >> > > > > This is a very good point. While building RPMs is not
> difficult,
> >> it
> >> > > does
> >> > > > > require time to setup the configuration and test the builds. I
> >> will
> >> > > > > include time for this in the proposal.
> >> > > > >
> >> > > >
> >> > > > I suppose I am really thinking about are scripts for starting and
> >> > > stopping
> >> > > > Accumulo. For an RPM to be useful it should provide good
> >> /etc/init.d
> >> > > > scripts for starting and stopping Accumulo processes. I am not
> >> sure
> >> > > what
> >> > > > the status of these scripts is in the various RPMs built by
> >> Accumulo.
> >> > > This
> >> > > > may not even be a concern, I am not sure how Ambari starts and
> stop
> >> > > > services. If you have not already, its something to consider
> when
> >> > > > scheduling.
> >> > > >
> >> > > >
> >> > > > >
> >> > > > > Thanks again, Keith.
> >> > > > >
> >> > > > > Andres
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Thu, Apr 25, 2013 at 10:54 AM, Keith Turner <
> keith@deenlo.com>
> >> > > wrote:
> >> > > > >
> >> > > > > > About the REST API, I meant to ask if this could be used from
> >> the
> >> > > > command
> >> > > > > > line currently? Or is too cumbersome?
> >> > > > > >
> >> > > > > >
> >> > > > > > On Thu, Apr 25, 2013 at 10:09 AM, Andres Danter <
> >> adanter@gmail.com>
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > > Yup. Looks like that's the way to go. Here is the link to
> >> the
> >> > > > > document.
> >> > > > > > > Let me know if you have any problems accessing it.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >>
> https://docs.google.com/file/d/0BxBY29P7A1RgVnZscGcycExDbEU/edit?usp=sharing
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Thu, Apr 25, 2013 at 9:51 AM, Keith Turner <
> >> keith@deenlo.com>
> >> > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Nothing. I suppose the mailing list is stripping it off.
> >> Could
> >> > > > post
> >> > > > > > it
> >> > > > > > > as
> >> > > > > > > > an attachment to the ticket or put it on google drive or
> >> dropbox
> >> > > > and
> >> > > > > > > send a
> >> > > > > > > > link.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Thu, Apr 25, 2013 at 9:44 AM, Andres Danter <
> >> > > adanter@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Weird. I see the attachment on the email I sent, which
> >> means
> >> > > > that
> >> > > > > it
> >> > > > > > > > > might have been stripped off. Here it is again. Let me
> >> know
> >> > > if
> >> > > > > you
> >> > > > > > > > still
> >> > > > > > > > > don't see it.
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > >
> >> > > > > > > > > Andres
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Thu, Apr 25, 2013 at 9:18 AM, Keith Turner <
> >> > > keith@deenlo.com>
> >> > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > >> was there supposed to be an attachment?
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > > > >> On Thu, Apr 25, 2013 at 2:19 AM, Andres Danter <
> >> > > > adanter@gmail.com
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > > >>
> >> > > > > > > > >> > Hi Billie,
> >> > > > > > > > >> >
> >> > > > > > > > >> > Here is a draft of my proposal. I've yet to write
> the
> >> > > > technical
> >> > > > > > > > >> approach
> >> > > > > > > > >> > section (have not idea what I'm going to put there)
> >> and the
> >> > > > > "about
> >> > > > > > > me"
> >> > > > > > > > >> > section. I definitely appreciate any feedback from
> >> you and
> >> > > > > anyone
> >> > > > > > > in
> >> > > > > > > > >> the
> >> > > > > > > > >> > development group.
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thank you,
> >> > > > > > > > >> >
> >> > > > > > > > >> > Andres
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >>
> >>
> >
> >
>