You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hbase.apache.org by Sean Busbey <bu...@cloudera.com> on 2015/07/05 01:01:31 UTC

Current status of 1.2.0RC0

Hi Folks!

I believe I've now worked through the logistics to put up the first RC for
1.2.0. At the moment I'm waiting on a Procedure V2 blocker[1], which I hope
means we'll be ready to go sometime Monday. Unless someone would like
otherwise, I'm planning to have a voting window of a week.

As a reminder, once RC0 goes up for a vote we'll be feature locked for the
1.2.0 release.


[1]: https://issues.apache.org/jira/browse/HBASE-14017

-- 
Sean

Re: Current status of 1.2.0RC0

Posted by Stack <st...@duboce.net>.

On Sun, Jul 5, 2015 at 11:56 AM, Andrew Purtell <ap...@apache.org> wrote:

> We haven't had a chance to test distributed log replay. Maybe later this
> year.
>
>
I'm trying a run with it now to get a basic measure. Will be back.

St.Ack


>
> On Sun, Jul 5, 2015 at 11:35 AM, Stack <st...@duboce.net> wrote:
>
> > On Sat, Jul 4, 2015 at 4:01 PM, Sean Busbey <bu...@cloudera.com> wrote:
> >
> > > Hi Folks!
> > >
> > > I believe I've now worked through the logistics to put up the first RC
> > for
> > > 1.2.0. At the moment I'm waiting on a Procedure V2 blocker[1], which I
> > hope
> > > means we'll be ready to go sometime Monday. Unless someone would like
> > > otherwise, I'm planning to have a voting window of a week.
> > >
> > > As a reminder, once RC0 goes up for a vote we'll be feature locked for
> > the
> > > 1.2.0 release.
> > >
> > >
> > > [1]: https://issues.apache.org/jira/browse/HBASE-14017
> >
> >
> > May I add HBASE-14012 to the above list Sean? (Almost done)
> >
> > 1.2.0 has Distributed Log Replay enabled by default. We good with this?
> > I've not done much testing with it enabled. Have others?
> >
> > 1.2.0 also has flush-by-store enabled by default. This has been tested a
> > bunch and looks pretty good to me.
> >
> > St.Ack
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: Current status of 1.2.0RC0

Posted by Andrew Purtell <ap...@apache.org>.

We haven't had a chance to test distributed log replay. Maybe later this
year.


On Sun, Jul 5, 2015 at 11:35 AM, Stack <st...@duboce.net> wrote:

> On Sat, Jul 4, 2015 at 4:01 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > Hi Folks!
> >
> > I believe I've now worked through the logistics to put up the first RC
> for
> > 1.2.0. At the moment I'm waiting on a Procedure V2 blocker[1], which I
> hope
> > means we'll be ready to go sometime Monday. Unless someone would like
> > otherwise, I'm planning to have a voting window of a week.
> >
> > As a reminder, once RC0 goes up for a vote we'll be feature locked for
> the
> > 1.2.0 release.
> >
> >
> > [1]: https://issues.apache.org/jira/browse/HBASE-14017
>
>
> May I add HBASE-14012 to the above list Sean? (Almost done)
>
> 1.2.0 has Distributed Log Replay enabled by default. We good with this?
> I've not done much testing with it enabled. Have others?
>
> 1.2.0 also has flush-by-store enabled by default. This has been tested a
> bunch and looks pretty good to me.
>
> St.Ack
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Current status of 1.2.0RC0

Posted by Sean Busbey <bu...@cloudera.com>.

On Jul 6, 2015 7:00 PM, "Enis Söztutar" <en...@gmail.com> wrote:
>
> I would put https://issues.apache.org/jira/browse/HBASE-13832 and
> HBASE-13415 also in the must-have list.
>

wfm. Please make sure they are marked blocker with a fix version of 1.2.0.

> For DLR, +1 for disable. I would not be comfortable with it going enabled
> by default unless it is tested throughly.
>

I'll start a DISCUSS thread to make sure any opposition has a chance to be
heard.

-- 
Sean

Re: Current status of 1.2.0RC0

Posted by Matteo Bertozzi <th...@gmail.com>.

the current proc-v2 does not do anything magic,
and does not do communication from master to RSs,
so there is nothing related to rolling upgrades.
and there is a test that verifies client compatibility.
you can do rolling upgrade between 0.98 to 1.2+.
you can have mix matched client and server version
or master version and everything works as before.
of course you don't have the benefit of proc-v2 until your cluster is
updated.
(if something fails in the middle of a master operation you still have to
run hbck)


Matteo


On Thu, Jul 9, 2015 at 11:57 AM, Andrew Purtell <ap...@apache.org> wrote:

> Just to be clear, of course this doesn't work if pv2 isn't opt out, so
> that's the question I'm really asking - should we / can we do that?
>
> On Thu, Jul 9, 2015 at 11:55 AM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > > I'm concerned about a sever side hbase rolling upgrade where masters
> and
> > > rs's are different versions / settings.  E.g. Does a pv2 only master
> > > failover properly with a nonpv2 master in the presence of mixed version
> > > rs's.  Does the master failover test cover this situation?
> >
> > This is a valid concern certainly. How we handled this in the 0.98 era,
> > which admittedly is looser by intent, is say that rolling upgrade are
> > supported IF the server fleet does not toggle on any new feature until
> all
> > server instances have been upgraded. Would that work here? I think it's a
> > reasonable story.
> >
> >
> > On Thu, Jul 9, 2015 at 6:47 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> >
> >> Let's have some testing of this before we commit to this decision. I'd
> >> hate
> >> for us to be in a situation where reality doesn't jive with theory due
> to
> >> something self inflicted.  I also feel that removing well exercised code
> >> paths in minor versions seems risky. (No qualms for removing in major
> >> version)
> >>
> >> My main concern isn't hbase client to hbase server. I buy that.
> >>
> >> I'm concerned about a sever side hbase rolling upgrade where masters and
> >> rs's are different versions / settings.  E.g. Does a pv2 only master
> >> failover properly with a nonpv2 master in the presence of mixed version
> >> rs's.  Does the master failover test cover this situation?
> >>
> >> Jon
> >>
> >> On Monday, July 6, 2015, Enis Söztutar <enis@apache.org
> >> <javascript:_e(%7B%7D,'cvml','enis@apache.org');>> wrote:
> >>
> >> > >
> >> > >
> >> > > > Would not being able to opt out of this break rolling upgrade from
> >> 1.0
> >> > or
> >> > > 1.1?
> >> > >
> >> >
> >> > It should not (in theory). The client side does not need to know that
> >> the
> >> > operation is executed via proc v2. The HBaseAdmin class has the
> >> > compatibility layer to work with masters which know about proc v2 or
> >> not.
> >> > And if the client does not know about proc v2, it will still observe
> the
> >> > side affects (whether the tables regions are created in meta, etc) and
> >> work
> >> > as expected.
> >> >
> >> >
> >> > >
> >> > >
> >> > > > Enis
> >> > > >
> >> > > > On Sun, Jul 5, 2015 at 2:01 PM, Sean Busbey <bu...@cloudera.com>
> >> > wrote:
> >> > > >
> >> > > > > On Jul 5, 2015 1:36 PM, "Stack" <st...@duboce.net> wrote:
> >> > > > > >
> >> > > > > > On Sat, Jul 4, 2015 at 4:01 PM, Sean Busbey <
> >> busbey@cloudera.com>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Folks!
> >> > > > > > >
> >> > > > > > > I believe I've now worked through the logistics to put up
> the
> >> > first
> >> > > > RC
> >> > > > > for
> >> > > > > > > 1.2.0. At the moment I'm waiting on a Procedure V2
> blocker[1],
> >> > > which
> >> > > > I
> >> > > > > hope
> >> > > > > >
> >> > > > > >
> >> > > > > > May I add HBASE-14012 to the above list Sean? (Almost done)
> >> > > > > >
> >> > > > >
> >> > > > > Fine by me. Please make sure it is blocker priority with a fix
> >> > version
> >> > > of
> >> > > > > 1.2.0.
> >> > > > >
> >> > > > > > 1.2.0 has Distributed Log Replay enabled by default. We good
> >> with
> >> > > this?
> >> > > > > > I've not done much testing with it enabled. Have others?
> >> > > > > >
> >> > > > >
> >> > > > > I haven't yet. I figured during RC0 I'd try to hit it hard and
> >> then
> >> > > file
> >> > > > > tickets as needed.
> >> > > > >
> >> > > > > If we leave it on we'll need docs for how to do a rolling
> upgrade.
> >> > > > >
> >> > > > > > 1.2.0 also has flush-by-store enabled by default. This has
> been
> >> > > tested
> >> > > > a
> >> > > > > > bunch and looks pretty good to me.
> >> > > > > >
> >> > > > >
> >> > > > > Good to hear, this and can't-opt-out-procv2 are my other big
> >> > unknowns.
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > // Jonathan Hsieh (shay)
> >> > > // HBase Tech Lead, Software Engineer, Cloudera
> >> > > // jon@cloudera.com // @jmhsieh
> >> > >
> >> >
> >>
> >>
> >> --
> >> // Jonathan Hsieh (shay)
> >> // HBase Tech Lead, Software Engineer, Cloudera
> >> // jon@cloudera.com // @jmhsieh
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: Current status of 1.2.0RC0

Posted by Enis Söztutar <en...@apache.org>.

>
> >> Let's have some testing of this before we commit to this decision. I'd
> >> hate
> >> for us to be in a situation where reality doesn't jive with theory due
> to
> >> something self inflicted.  I also feel that removing well exercised code
> >> paths in minor versions seems risky. (No qualms for removing in major
> >> version)
>

That is why I suggested that we keep the handlers in branch-1.1 so that in
case something serious comes up (especially related to customer support
environments) that at least we can go back to previous handler logic and
sideline the issues. We do not need to (and should not) keep handlers
around in branch-1 from a compatibility point of view. Whether to keep it
as a backup option to be able to sleep better at night is debatable.
Maintaining handlers in the code base is not difficult, but we need to
re-add them back since in branch-1 they are already removed (see release
notes for https://issues.apache.org/jira/browse/HBASE-13469). I think we
should never give the option of using handlers, only except for guarding
against problems in procv2 without any workaround.

We have been running all sorts of IT tests including
IntegrationTestDDLMasterFailover, and some other master failover tests as
well as rolling upgrade from 0.98.4 (HDP-2.2) versions, and all of the
discovered issues have been fixed (see subtasks of HBASE-12439). We
consider proc v2 to be pretty stable thanks to Matteo and Stephen's work.

For rolling upgrades between procv2 masters and non-procv2 masters, RU
should work. These are independent of each other with minimal shared state
(except for actual hdfs and zk data structures). If you have a master
failure during create table for example, the outcome was not defined. In
that sense it does not change the semantics.

Enis


> >>
> >> My main concern isn't hbase client to hbase server. I buy that.
> >>
> >> I'm concerned about a sever side hbase rolling upgrade where masters and
> >> rs's are different versions / settings.  E.g. Does a pv2 only master
> >> failover properly with a nonpv2 master in the presence of mixed version
> >> rs's.  Does the master failover test cover this situation?
> >>
> >> Jon
> >>
> >> On Monday, July 6, 2015, Enis Söztutar <enis@apache.org
> >> <javascript:_e(%7B%7D,'cvml','enis@apache.org');>> wrote:
> >>
> >> > >
> >> > >
> >> > > > Would not being able to opt out of this break rolling upgrade from
> >> 1.0
> >> > or
> >> > > 1.1?
> >> > >
> >> >
> >> > It should not (in theory). The client side does not need to know that
> >> the
> >> > operation is executed via proc v2. The HBaseAdmin class has the
> >> > compatibility layer to work with masters which know about proc v2 or
> >> not.
> >> > And if the client does not know about proc v2, it will still observe
> the
> >> > side affects (whether the tables regions are created in meta, etc) and
> >> work
> >> > as expected.
> >> >
> >> >
> >> > >
> >> > >
> >> > > > Enis
> >> > > >
> >> > > > On Sun, Jul 5, 2015 at 2:01 PM, Sean Busbey <bu...@cloudera.com>
> >> > wrote:
> >> > > >
> >> > > > > On Jul 5, 2015 1:36 PM, "Stack" <st...@duboce.net> wrote:
> >> > > > > >
> >> > > > > > On Sat, Jul 4, 2015 at 4:01 PM, Sean Busbey <
> >> busbey@cloudera.com>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Folks!
> >> > > > > > >
> >> > > > > > > I believe I've now worked through the logistics to put up
> the
> >> > first
> >> > > > RC
> >> > > > > for
> >> > > > > > > 1.2.0. At the moment I'm waiting on a Procedure V2
> blocker[1],
> >> > > which
> >> > > > I
> >> > > > > hope
> >> > > > > >
> >> > > > > >
> >> > > > > > May I add HBASE-14012 to the above list Sean? (Almost done)
> >> > > > > >
> >> > > > >
> >> > > > > Fine by me. Please make sure it is blocker priority with a fix
> >> > version
> >> > > of
> >> > > > > 1.2.0.
> >> > > > >
> >> > > > > > 1.2.0 has Distributed Log Replay enabled by default. We good
> >> with
> >> > > this?
> >> > > > > > I've not done much testing with it enabled. Have others?
> >> > > > > >
> >> > > > >
> >> > > > > I haven't yet. I figured during RC0 I'd try to hit it hard and
> >> then
> >> > > file
> >> > > > > tickets as needed.
> >> > > > >
> >> > > > > If we leave it on we'll need docs for how to do a rolling
> upgrade.
> >> > > > >
> >> > > > > > 1.2.0 also has flush-by-store enabled by default. This has
> been
> >> > > tested
> >> > > > a
> >> > > > > > bunch and looks pretty good to me.
> >> > > > > >
> >> > > > >
> >> > > > > Good to hear, this and can't-opt-out-procv2 are my other big
> >> > unknowns.
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > // Jonathan Hsieh (shay)
> >> > > // HBase Tech Lead, Software Engineer, Cloudera
> >> > > // jon@cloudera.com // @jmhsieh
> >> > >
> >> >
> >>
> >>
> >> --
> >> // Jonathan Hsieh (shay)
> >> // HBase Tech Lead, Software Engineer, Cloudera
> >> // jon@cloudera.com // @jmhsieh
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: Current status of 1.2.0RC0

Posted by Andrew Purtell <ap...@apache.org>.

Just to be clear, of course this doesn't work if pv2 isn't opt out, so
that's the question I'm really asking - should we / can we do that?

On Thu, Jul 9, 2015 at 11:55 AM, Andrew Purtell <ap...@apache.org> wrote:

> > I'm concerned about a sever side hbase rolling upgrade where masters and
> > rs's are different versions / settings.  E.g. Does a pv2 only master
> > failover properly with a nonpv2 master in the presence of mixed version
> > rs's.  Does the master failover test cover this situation?
>
> This is a valid concern certainly. How we handled this in the 0.98 era,
> which admittedly is looser by intent, is say that rolling upgrade are
> supported IF the server fleet does not toggle on any new feature until all
> server instances have been upgraded. Would that work here? I think it's a
> reasonable story.
>
>
> On Thu, Jul 9, 2015 at 6:47 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
>> Let's have some testing of this before we commit to this decision. I'd
>> hate
>> for us to be in a situation where reality doesn't jive with theory due to
>> something self inflicted.  I also feel that removing well exercised code
>> paths in minor versions seems risky. (No qualms for removing in major
>> version)
>>
>> My main concern isn't hbase client to hbase server. I buy that.
>>
>> I'm concerned about a sever side hbase rolling upgrade where masters and
>> rs's are different versions / settings.  E.g. Does a pv2 only master
>> failover properly with a nonpv2 master in the presence of mixed version
>> rs's.  Does the master failover test cover this situation?
>>
>> Jon
>>
>> On Monday, July 6, 2015, Enis Söztutar <enis@apache.org
>> <javascript:_e(%7B%7D,'cvml','enis@apache.org');>> wrote:
>>
>> > >
>> > >
>> > > > Would not being able to opt out of this break rolling upgrade from
>> 1.0
>> > or
>> > > 1.1?
>> > >
>> >
>> > It should not (in theory). The client side does not need to know that
>> the
>> > operation is executed via proc v2. The HBaseAdmin class has the
>> > compatibility layer to work with masters which know about proc v2 or
>> not.
>> > And if the client does not know about proc v2, it will still observe the
>> > side affects (whether the tables regions are created in meta, etc) and
>> work
>> > as expected.
>> >
>> >
>> > >
>> > >
>> > > > Enis
>> > > >
>> > > > On Sun, Jul 5, 2015 at 2:01 PM, Sean Busbey <bu...@cloudera.com>
>> > wrote:
>> > > >
>> > > > > On Jul 5, 2015 1:36 PM, "Stack" <st...@duboce.net> wrote:
>> > > > > >
>> > > > > > On Sat, Jul 4, 2015 at 4:01 PM, Sean Busbey <
>> busbey@cloudera.com>
>> > > > wrote:
>> > > > > >
>> > > > > > > Hi Folks!
>> > > > > > >
>> > > > > > > I believe I've now worked through the logistics to put up the
>> > first
>> > > > RC
>> > > > > for
>> > > > > > > 1.2.0. At the moment I'm waiting on a Procedure V2 blocker[1],
>> > > which
>> > > > I
>> > > > > hope
>> > > > > >
>> > > > > >
>> > > > > > May I add HBASE-14012 to the above list Sean? (Almost done)
>> > > > > >
>> > > > >
>> > > > > Fine by me. Please make sure it is blocker priority with a fix
>> > version
>> > > of
>> > > > > 1.2.0.
>> > > > >
>> > > > > > 1.2.0 has Distributed Log Replay enabled by default. We good
>> with
>> > > this?
>> > > > > > I've not done much testing with it enabled. Have others?
>> > > > > >
>> > > > >
>> > > > > I haven't yet. I figured during RC0 I'd try to hit it hard and
>> then
>> > > file
>> > > > > tickets as needed.
>> > > > >
>> > > > > If we leave it on we'll need docs for how to do a rolling upgrade.
>> > > > >
>> > > > > > 1.2.0 also has flush-by-store enabled by default. This has been
>> > > tested
>> > > > a
>> > > > > > bunch and looks pretty good to me.
>> > > > > >
>> > > > >
>> > > > > Good to hear, this and can't-opt-out-procv2 are my other big
>> > unknowns.
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > // Jonathan Hsieh (shay)
>> > > // HBase Tech Lead, Software Engineer, Cloudera
>> > > // jon@cloudera.com // @jmhsieh
>> > >
>> >
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // HBase Tech Lead, Software Engineer, Cloudera
>> // jon@cloudera.com // @jmhsieh
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Current status of 1.2.0RC0

Posted by Andrew Purtell <ap...@apache.org>.

> I'm concerned about a sever side hbase rolling upgrade where masters and
> rs's are different versions / settings.  E.g. Does a pv2 only master
> failover properly with a nonpv2 master in the presence of mixed version
> rs's.  Does the master failover test cover this situation?

This is a valid concern certainly. How we handled this in the 0.98 era,
which admittedly is looser by intent, is say that rolling upgrade are
supported IF the server fleet does not toggle on any new feature until all
server instances have been upgraded. Would that work here? I think it's a
reasonable story.


On Thu, Jul 9, 2015 at 6:47 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:

> Let's have some testing of this before we commit to this decision. I'd hate
> for us to be in a situation where reality doesn't jive with theory due to
> something self inflicted.  I also feel that removing well exercised code
> paths in minor versions seems risky. (No qualms for removing in major
> version)
>
> My main concern isn't hbase client to hbase server. I buy that.
>
> I'm concerned about a sever side hbase rolling upgrade where masters and
> rs's are different versions / settings.  E.g. Does a pv2 only master
> failover properly with a nonpv2 master in the presence of mixed version
> rs's.  Does the master failover test cover this situation?
>
> Jon
>
> On Monday, July 6, 2015, Enis Söztutar <enis@apache.org
> <javascript:_e(%7B%7D,'cvml','enis@apache.org');>> wrote:
>
> > >
> > >
> > > > Would not being able to opt out of this break rolling upgrade from
> 1.0
> > or
> > > 1.1?
> > >
> >
> > It should not (in theory). The client side does not need to know that the
> > operation is executed via proc v2. The HBaseAdmin class has the
> > compatibility layer to work with masters which know about proc v2 or not.
> > And if the client does not know about proc v2, it will still observe the
> > side affects (whether the tables regions are created in meta, etc) and
> work
> > as expected.
> >
> >
> > >
> > >
> > > > Enis
> > > >
> > > > On Sun, Jul 5, 2015 at 2:01 PM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > > >
> > > > > On Jul 5, 2015 1:36 PM, "Stack" <st...@duboce.net> wrote:
> > > > > >
> > > > > > On Sat, Jul 4, 2015 at 4:01 PM, Sean Busbey <busbey@cloudera.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hi Folks!
> > > > > > >
> > > > > > > I believe I've now worked through the logistics to put up the
> > first
> > > > RC
> > > > > for
> > > > > > > 1.2.0. At the moment I'm waiting on a Procedure V2 blocker[1],
> > > which
> > > > I
> > > > > hope
> > > > > >
> > > > > >
> > > > > > May I add HBASE-14012 to the above list Sean? (Almost done)
> > > > > >
> > > > >
> > > > > Fine by me. Please make sure it is blocker priority with a fix
> > version
> > > of
> > > > > 1.2.0.
> > > > >
> > > > > > 1.2.0 has Distributed Log Replay enabled by default. We good with
> > > this?
> > > > > > I've not done much testing with it enabled. Have others?
> > > > > >
> > > > >
> > > > > I haven't yet. I figured during RC0 I'd try to hit it hard and then
> > > file
> > > > > tickets as needed.
> > > > >
> > > > > If we leave it on we'll need docs for how to do a rolling upgrade.
> > > > >
> > > > > > 1.2.0 also has flush-by-store enabled by default. This has been
> > > tested
> > > > a
> > > > > > bunch and looks pretty good to me.
> > > > > >
> > > > >
> > > > > Good to hear, this and can't-opt-out-procv2 are my other big
> > unknowns.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > // Jonathan Hsieh (shay)
> > > // HBase Tech Lead, Software Engineer, Cloudera
> > > // jon@cloudera.com // @jmhsieh
> > >
> >
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)