You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "张铎(Duo Zhang)" <pa...@gmail.com> on 2023/12/01 01:27:40 UTC
Re: Time for 2.5.6 and 2.6.0.
Thanks Andrew. I recall that there was a discussion around this topic,
but I had not followed up to the end so I did not know the discussion
result...
So at least we could add a tool in create-release for cutting release branches.
Andrew Purtell <an...@gmail.com> 于2023年12月1日周五 01:16写道：
>
> Maven automates the change so the human does not edit the POM files by hand. A PR to check this is wasteful of people’s time because a reviewer is just signing off on a change made by a tool and won’t check the details (this is human nature) and the RM is sitting around for who knows how long waiting for the sign off (this is the actual problem) and as you know the process for making a release is very lengthy already.
>
> Apache release requirements forbid use of Jenkins to generate release artifacts. There are various conversations about this in the archives of infrastructure@ or members@ or perhaps someone like Nick more directly involved with build infrastructure has more context. Something to do with automated pipelines can be hacked and having humans doing enough by hand that they can really attest to the results provides more assurance.
>
> PRs are valuable and necessary most of the time because the changes are made by humans and are nontrivial and hugely benefit from review by another person. But PRs for signing off on changes made by tools are not.
>
> > On Nov 29, 2023, at 11:20 PM, 张铎 <pa...@gmail.com> wrote:
> >
> > That's the problem! Now we use $revision for better command line
> > support when releasing 2.5.x and 2.5.x-hadoop3, so the command for
> > bumping version is
> >
> > mvn versions:set-property -Dproperty=revision -DnewVersion=<newversion>
> >
> > I've been thinking of introducing a jenkins job for creating release
> > for a long time, but haven't gotten enough time to do it since it is
> > not easy for me as I'm not a jenkins expert...
> >
> > Andrew Purtell <ap...@apache.org> 于2023年11月30日周四 10:45写道：
> >>
> >> It seems silly to do a review just for a change that modifies the project
> >> version. I mean, it's trivial to "mvn versions:set
> >> -DnewVersion=<newversion>", and surely RMs can be trusted to do this and
> >> not mess it up. You might think the PR is fine, but it can introduce hours
> >> if not days of delay in the release process, which takes long enough
> >> already. Anyway glad we agree results produced by release tools don't
> >> require signoff.
> >>
> >>> On Wed, Nov 29, 2023 at 5:13 PM 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> >>>
> >>> Just for checking there is no mistake :)
> >>>
> >>> And if we can formalize the cutting release branch related operations
> >>> in a release tool, I do not think we need the signoff, just running
> >>> the automatic tool is fine, but now it still requires manual
> >>> operations, so I think having a double check is better.
> >>>
> >>> Thanks.
> >>>
> >>> Andrew Purtell <ap...@apache.org> 于2023年11月30日周四 02:45写道：
> >>>>
> >>>> I don't think RMs should need to get PR approvals for version changes in
> >>>> the POMs. The create-release script normally does this automatically
> >>>> anyway. Sure, this is renumbering branch-2, not a release branch, but who
> >>>> is going to object?
> >>>>
> >>>> On Wed, Nov 29, 2023 at 5:38 AM 张铎(Duo Zhang) <pa...@gmail.com>
> >>> wrote:
> >>>>
> >>>>> Better to open an umbrella issue for tracking the task, and open a
> >>>>> sub-task for each step.
> >>>>>
> >>>>> For bumping version to 2.7.0-SNAPSHOT, still better to open an PR as
> >>>>> it is a code commit.
> >>>>>
> >>>>> Others are all good. Thanks Byran for taking care of this!
> >>>>>
> >>>>> Bryan Beaudreault <bb...@apache.org> 于2023年11月29日周三 21:24写道：
> >>>>>>
> >>>>>> Here's my planned approach. Please let me know if this looks good:
> >>>>>>
> >>>>>> # Create new branch
> >>>>>> git checkout branch-2
> >>>>>> git pull upstream branch-2
> >>>>>> git status # check all is good
> >>>>>> git checkout -b branch-2.6
> >>>>>> git push upstream branch-2.6
> >>>>>>
> >>>>>> # Update snapshot version on branch-2
> >>>>>> git checkout branch-2
> >>>>>> Edit root pom.xml, change properties.revision to 2.7.0-SNAPSHOT
> >>>>>> git commit, "Renumber to 2.7.0-SNAPSHOT after branching branch-2.6"
> >>> with
> >>>>>> self-sign-off
> >>>>>> git push upstream branch-2
> >>>>>>
> >>>>>> After that I will start looking through JIRA for fixVersion 2.6.0
> >>> issues,
> >>>>>> to make sure there's no inconsistencies. *Is there anything I need
> >>> to do
> >>>>> to
> >>>>>> set up nightly builds for the new branch? *We should probably let
> >>> nightly
> >>>>>> builds run for a few days before starting RC.
> >>>>>>
> >>>>>> Prior to starting the RC process, I will try deploying the newly
> >>> minted
> >>>>>> 2.6-snapshot onto a test cluster in my environment. I notice that
> >>> Andrew
> >>>>>> did a bunch of load and chaos testing for the 2.5.0 release. I will
> >>> try
> >>>>>> running these locally, but we have not ever run ITBLL, etc in my
> >>>>> company's
> >>>>>> environment so that might take some time to set up.
> >>>>>>
> >>>>>> On Wed, Nov 22, 2023 at 8:50 AM Bryan Beaudreault <
> >>>>> bbeaudreault@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Sorry for the delay. I just sent a new thread to the list to alert
> >>> to
> >>>>>>> creating branch-2.6 next week.
> >>>>>>>
> >>>>>>> On Tue, Nov 21, 2023 at 3:35 AM 张铎(Duo Zhang) <
> >>> palomino219@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Bump again :)
> >>>>>>>>
> >>>>>>>> Are we ready to cut branch-2.6?
> >>>>>>>>
> >>>>>>>> Bryan Beaudreault <bb...@apache.org> 于2023年10月26日周四
> >>> 21:22写道：
> >>>>>>>>>
> >>>>>>>>> Just as an update here, I plan to do this but have been busy
> >>> with
> >>>>>>>> internal
> >>>>>>>>> work. I'll try to carve out time in the next 1-2 weeks.
> >>>>>>>>>
> >>>>>>>>> On Thu, Oct 19, 2023 at 8:50 PM 张铎(Duo Zhang) <
> >>>>> palomino219@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Just send a notice to the dev mailing list, and then use the
> >>>>> normal
> >>>>>>>>>> git way to create a new branch called branch-2.6 from
> >>> branch-2,
> >>>>> push
> >>>>>>>>>> it to github, done :)
> >>>>>>>>>>
> >>>>>>>>>> Bryan Beaudreault <bb...@apache.org> 于2023年10月20日周五
> >>>>> 07:53写道：
> >>>>>>>>>>>
> >>>>>>>>>>> I can give it a shot :)
> >>>>>>>>>>>
> >>>>>>>>>>> I’ve started looking at the release docs, but don’t see any
> >>>>> info on
> >>>>>>>> how
> >>>>>>>>>> to
> >>>>>>>>>>> create a new release branch. Is there a process for that?
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Oct 19, 2023 at 1:55 PM Andrew Purtell <
> >>>>> apurtell@apache.org
> >>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> I second this nomination!
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Oct 19, 2023 at 6:39 AM 张铎(Duo Zhang) <
> >>>>>>>> palomino219@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> OK, good. So we are good to go I think.
> >>>>>>>>>>>>> And we need a release manager for 2.6.x, do you have any
> >>>>>>>> interest in
> >>>>>>>>>>>>> being a release manager for 2.6.x? :)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Bryan Beaudreault <bb...@apache.org>
> >>> 于2023年10月19日周四
> >>>>>>>> 21:22写道：
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hello. I'm not aware of any major blockers at the
> >>> moment.
> >>>>> In
> >>>>>>>> fact
> >>>>>>>>>> we
> >>>>>>>>>>>>>> recently finished our migration and are fully using
> >>> 2.6's
> >>>>> TLS
> >>>>>>>> in
> >>>>>>>>>> all of
> >>>>>>>>>>>>> our
> >>>>>>>>>>>>>> production.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Oct 19, 2023 at 12:07 AM 张铎(Duo Zhang) <
> >>>>>>>>>> palomino219@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Bryan, do we still have any blocker issues for
> >>>>> cutting
> >>>>>>>>>> branch-2.6
> >>>>>>>>>>>>>>> and starting to release 2.6.0?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Andrew Purtell <ap...@apache.org> 于2023年9月26日周二
> >>>>> 01:02写道：
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks Duo. Following up on HBASE-28042.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sun, Sep 24, 2023 at 7:53 PM 张铎(Duo Zhang) <
> >>>>>>>>>>>> palomino219@gmail.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Some updates here.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The tcnative module is almost done, only some
> >>> small
> >>>>>>>> concerns
> >>>>>>>>>> on
> >>>>>>>>>>>>> how to
> >>>>>>>>>>>>>>>>> deal with the license and notice files.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> There is still a blocker issue HBASE-28081, but
> >>>>> anyway
> >>>>>>>> we
> >>>>>>>>>> have
> >>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>> found the root cause is HBASE-28042, so I think
> >>> it
> >>>>> will
> >>>>>>>> be
> >>>>>>>>>> fixed
> >>>>>>>>>>>>> soon.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> All other issues have been resolved.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Andrew Purtell <ap...@apache.org>
> >>> 于2023年9月22日周五
> >>>>>>>> 02:21写道：
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Both sound good, thanks.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Tue, Sep 19, 2023 at 7:46 PM 张铎(Duo Zhang)
> >>> <
> >>>>>>>>>>>>> palomino219@gmail.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> We'd better get a new thirdparty release
> >>> first,
> >>>>>>>> update
> >>>>>>>>>>>>> branch-2 to
> >>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>>> the new thirdparty release, and then make
> >>> the
> >>>>> 2.6.0
> >>>>>>>>>> release,
> >>>>>>>>>>>>> since
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> new netty release includes the improvements
> >>>>>>>> contributed
> >>>>>>>>>> by
> >>>>>>>>>>>>> Bryan.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> https://github.com/netty/netty/pull/13551
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> The current blocking issues are
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> HBASE-28075, where we want to shade netty
> >>>>> tcnative
> >>>>>>>> as
> >>>>>>>>>> well,
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>> Bryan
> >>>>>>>>>>>>>>>>>>> said he already had a workable solution.
> >>>>>>>>>>>>>>>>>>> HBASE-22138, where we want to also
> >>> relocated the
> >>>>>>>> proto
> >>>>>>>>>> files
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>>> shaded
> >>>>>>>>>>>>>>>>>>> protobuf, there is no technical problem now
> >>> and
> >>>>> I've
> >>>>>>>>>> opened a
> >>>>>>>>>>>>> PR
> >>>>>>>>>>>>>>> using
> >>>>>>>>>>>>>>>>>>> the 4.1.5-SNAPSHOT version of
> >>> hbase-thirdparty,
> >>>>>>>>>>>>>>>>>>> https://github.com/apache/hbase/pull/5411.
> >>>>> There is
> >>>>>>>>>> another
> >>>>>>>>>>>>>>> problem
> >>>>>>>>>>>>>>>>>>> that we used to exclude CanIgnoreReturnValue
> >>>>>>>> annotation
> >>>>>>>>>> to
> >>>>>>>>>>>>> ignore
> >>>>>>>>>>>>>>> some
> >>>>>>>>>>>>>>>>>>> error prone errors, but I think we should
> >>> add it
> >>>>>>>> back
> >>>>>>>>>> and fix
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> error prone errors, instead of ignoring it,
> >>>>> should
> >>>>>>>> not be
> >>>>>>>>>>>> very
> >>>>>>>>>>>>>>> hard to
> >>>>>>>>>>>>>>>>>>> do.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> All the above issues have a clear path to
> >>>>> resolve,
> >>>>>>>> so I
> >>>>>>>>>> think
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>> not delay too much.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Bryan Beaudreault <bb...@gmail.com>
> >>>>>>>> 于2023年9月20日周三
> >>>>>>>>>>>>> 02:51写道：
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I hate to keep delaying, but it might be
> >>> nice
> >>>>> to
> >>>>>>>> wait
> >>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>
> >>>>> https://issues.apache.org/jira/browse/HBASE-28065.
> >>>>>>>> It
> >>>>>>>>>> was
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>> pretty
> >>>>>>>>>>>>>>>>> big
> >>>>>>>>>>>>>>>>>>> (if
> >>>>>>>>>>>>>>>>>>>> long standing) bug for us, and given the
> >>>>> solution
> >>>>>>>> is
> >>>>>>>>>> ready
> >>>>>>>>>>>> (
> >>>>>>>>>>>>>>>>>>>> https://github.com/apache/hbase/pull/5384)
> >>> we
> >>>>>>>> might as
> >>>>>>>>>>>> well
> >>>>>>>>>>>>> get
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> our
> >>>>>>>>>>>>>>>>>>>> next releases.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Tue, Sep 19, 2023 at 1:37 PM Andrew
> >>>>> Purtell <
> >>>>>>>>>>>>>>> apurtell@apache.org>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks Duo.
> >>>>>>>>>>>>>>>>>>>>> I do not need help.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I have been waiting to ensure more
> >>> should do
> >>>>>>>> items
> >>>>>>>>>> do not
> >>>>>>>>>>>>>>> appear.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I think we are good now?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2023 at 12:31 AM 张铎(Duo
> >>>>> Zhang) <
> >>>>>>>>>>>>>>>>> palomino219@gmail.com>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> HBASE-28061 has been merged.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Do you need any help on the release
> >>>>> processing
> >>>>>>>>>> Andrew?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I could offer my help.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> 张铎(Duo Zhang) <pa...@gmail.com>
> >>>>>>>>>> 于2023年9月14日周四
> >>>>>>>>>>>>> 15:55写道：
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> If you have not cut the RC yet,
> >>> please
> >>>>> also
> >>>>>>>>>> include
> >>>>>>>>>>>>>>>>> HBASE-28061 in
> >>>>>>>>>>>>>>>>>>>>>>> 2.5.6, as it could solve the
> >>> problem for
> >>>>>>>> running
> >>>>>>>>>> pre
> >>>>>>>>>>>>>>> compiled
> >>>>>>>>>>>>>>>>> 2.5.x
> >>>>>>>>>>>>>>>>>>>>>>> tarball and also our published
> >>>>> artifacts in
> >>>>>>>> maven
> >>>>>>>>>>>> repo
> >>>>>>>>>>>>>>> against
> >>>>>>>>>>>>>>>>>>> hadoop
> >>>>>>>>>>>>>>>>>>>>>>> 3.3.x.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> The hive community needs this as
> >>> they
> >>>>> want
> >>>>>>>> to
> >>>>>>>>>> upgrade
> >>>>>>>>>>>>>>> hbase and
> >>>>>>>>>>>>>>>>>>> hadoop
> >>>>>>>>>>>>>>>>>>>>>>> dependencies.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Andrew Purtell <apurtell@apache.org
> >>>>
> >>>>>>>>>> 于2023年8月30日周三
> >>>>>>>>>>>>>>> 06:48写道：
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> I am back from vacation. Slowly
> >>>>> clearing
> >>>>>>>> my
> >>>>>>>>>> inbox.
> >>>>>>>>>>>>> 1,545
> >>>>>>>>>>>>>>>>> unread,
> >>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>>> a while.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> It seems we are ready to cut
> >>>>> branch-2.6
> >>>>>>>> for
> >>>>>>>>>>>>> stabilizing
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> 2.6.0.
> >>>>>>>>>>>>>>>>>>>>> Any
> >>>>>>>>>>>>>>>>>>>>>>>> volunteers to RM 2.6.0? Otherwise
> >>> I'm
> >>>>>>>> happy to
> >>>>>>>>>> do
> >>>>>>>>>>>> it,
> >>>>>>>>>>>>>>>>> starting
> >>>>>>>>>>>>>>>>>>> next
> >>>>>>>>>>>>>>>>>>>>>> week,
> >>>>>>>>>>>>>>>>>>>>>>>> probably 9/5 or 9/6.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Some good changes have landed in
> >>>>>>>> branch-2.5.
> >>>>>>>>>> RC0
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>> 2.5.6 by
> >>>>>>>>>>>>>>>>>>> Friday
> >>>>>>>>>>>>>>>>>>>>>> 9/1.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Unrest, ignorance distilled, nihilistic
> >>> imbeciles
> >>>>> -
> >>>>>>>>>>>>>>>>>> It's what we’ve earned
> >>>>>>>>>>>>>>>>>> Welcome, apocalypse, what’s taken you so long?
> >>>>>>>>>>>>>>>>>> Bring us the fitting end that we’ve been
> >>> counting
> >>>>> on
> >>>>>>>>>>>>>>>>>> - A23, Welcome, Apocalypse
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Unrest, ignorance distilled, nihilistic imbeciles
> >>> -
> >>>>>>>>>>>>>>>> It's what we’ve earned
> >>>>>>>>>>>>>>>> Welcome, apocalypse, what’s taken you so long?
> >>>>>>>>>>>>>>>> Bring us the fitting end that we’ve been counting
> >>> on
> >>>>>>>>>>>>>>>> - A23, Welcome, Apocalypse
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Best regards,
> >>>>>>>>>>>> Andrew
> >>>>>>>>>>>>
> >>>>>>>>>>>> Unrest, ignorance distilled, nihilistic imbeciles -
> >>>>>>>>>>>> It's what we’ve earned
> >>>>>>>>>>>> Welcome, apocalypse, what’s taken you so long?
> >>>>>>>>>>>> Bring us the fitting end that we’ve been counting on
> >>>>>>>>>>>> - A23, Welcome, Apocalypse
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best regards,
> >>>> Andrew
> >>>>
> >>>> Unrest, ignorance distilled, nihilistic imbeciles -
> >>>> It's what we’ve earned
> >>>> Welcome, apocalypse, what’s taken you so long?
> >>>> Bring us the fitting end that we’ve been counting on
> >>>> - A23, Welcome, Apocalypse
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >> Andrew
> >>
> >> Unrest, ignorance distilled, nihilistic imbeciles -
> >> It's what we’ve earned
> >> Welcome, apocalypse, what’s taken you so long?
> >> Bring us the fitting end that we’ve been counting on
> >> - A23, Welcome, Apocalypse