You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Kevin Burton <bu...@spinn3r.com> on 2015/05/02 21:23:33 UTC
Setting expireMessagesPeriod=0 for advisory topics improved GC
performance by 2x..
I’m doing a bunch of performance analysis of ActiveMQ this weekend to see
if I can improve queue creation and destruction time. The good news is
that there are a lot of areas of optimization.
It looks like one is that advisory topics are created with the default
expireMessagesPeriod (which is 30000). However, no advisory messages are
produced with TTLs.
I believe this is done in AdvisoryBroker.fireAdvisory
… looking at this code, a TTL isn’t set. So I think that it would be a
good idea to enable expireMessagesPeriod=0 for advisory topics.
Here’s the code if you’re interested:
http://pastebin.com/i3hXDGEW
Looks like the following policyEntry works as a proof of concept.
<policyEntry topic=">"
prioritizedMessages="false"
useCache="false"
maxBrowsePageSize="2147483647"
expireMessagesPeriod="0">
</policyEntry>
(I am going to constrain it for advisory topics though).
In my test time time for GCing all my queues went from 45s to 17s. It’s
MUCH higher than this under production load though due threading issues.
Are there any potential issues I might be missing here?
If this works I might try to submit a patch to disable expireMessagesPeriod
by default for advisory topics.
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Setting expireMessagesPeriod=0 for advisory topics improved GC
performance by 2x..
Posted by Tim Bain <tb...@alumni.duke.edu>.
Looks good to me.
On Mon, May 11, 2015 at 12:52 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> > I hadn't expected purge() to call heapify(), so I was just expecting O(N)
> > runtime for the actual removal. The comments claim the runtime of the
> > purge operation is O(N + ClogN) where C is the number of deleted elements
> > (which would make it O(N) when C is 1, but looking at the code I think
> it's
> > actually O(N + ((N-C)/2)log((N-C)/2)), which is O(NlogN) when C is 1. So
> > it's not O(N^2), but it's also definitely not O(N) despite the comment's
> > claim.
> >
> >
> Yes. I think this is a more appropriate claim. But in my experience it
> seems to grow larger than NlogN but perhaps there are secondary issues here
> like L1/L2 cache hits that dramatically slow it down
>
>
> > Now I just have it essentially flag the timer task as clear and it will
> go
> > > through the timer queue and not get executed.
> > >
> >
> > Sounds fine, as long as "flag the timer task as clear" also means "call
> > TimerTask.cancel(), so that the task's state will be TimerTask.CANCELLED
> so
> > it'll be removed the next time it comes to the top of the queue in
> > Timer.mainLoop(), in addition to any WeakReference stuff we may do". If
> > not, you're going to have a lot of no-op TimerTasks that stick around
> > forever, so 1) your value of N will grow over time, so even O(N)
> operations
> > will take longer over time, 2) there's still an expensive thread context
> > switch even if the task is a nominal no-op, and 3) eventually you could
> OOM
> > due to the memory those no-op TimerTasks take. Does that match what
> you've
> > implemented?
> >
> >
> Yes. here’s the new cancel method:
>
> https://gist.github.com/burtonator/6fe3ee640f5f1b27caa2
>
> I call task .cancel() then .clear() (which clears the reference).
>
> and here’s the new SchedulerTimerTask
>
> https://gist.github.com/burtonator/be4bd7b39c250b2b1a0c
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
Re: Setting expireMessagesPeriod=0 for advisory topics improved GC
performance by 2x..
Posted by Kevin Burton <bu...@spinn3r.com>.
> I hadn't expected purge() to call heapify(), so I was just expecting O(N)
> runtime for the actual removal. The comments claim the runtime of the
> purge operation is O(N + ClogN) where C is the number of deleted elements
> (which would make it O(N) when C is 1, but looking at the code I think it's
> actually O(N + ((N-C)/2)log((N-C)/2)), which is O(NlogN) when C is 1. So
> it's not O(N^2), but it's also definitely not O(N) despite the comment's
> claim.
>
>
Yes. I think this is a more appropriate claim. But in my experience it
seems to grow larger than NlogN but perhaps there are secondary issues here
like L1/L2 cache hits that dramatically slow it down
> Now I just have it essentially flag the timer task as clear and it will go
> > through the timer queue and not get executed.
> >
>
> Sounds fine, as long as "flag the timer task as clear" also means "call
> TimerTask.cancel(), so that the task's state will be TimerTask.CANCELLED so
> it'll be removed the next time it comes to the top of the queue in
> Timer.mainLoop(), in addition to any WeakReference stuff we may do". If
> not, you're going to have a lot of no-op TimerTasks that stick around
> forever, so 1) your value of N will grow over time, so even O(N) operations
> will take longer over time, 2) there's still an expensive thread context
> switch even if the task is a nominal no-op, and 3) eventually you could OOM
> due to the memory those no-op TimerTasks take. Does that match what you've
> implemented?
>
>
Yes. here’s the new cancel method:
https://gist.github.com/burtonator/6fe3ee640f5f1b27caa2
I call task .cancel() then .clear() (which clears the reference).
and here’s the new SchedulerTimerTask
https://gist.github.com/burtonator/be4bd7b39c250b2b1a0c
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Setting expireMessagesPeriod=0 for advisory topics improved GC
performance by 2x..
Posted by Tim Bain <tb...@alumni.duke.edu>.
On Sat, May 9, 2015 at 9:41 AM, Kevin Burton <bu...@spinn3r.com> wrote:
> > I wasn't at all clear why changing the frequency (to "never") of message
> > expiration checks would affect the performance of destination GC
> operations
> > when the number of queues is large. Is that due to synchronization
> between
> > those two operations, or thread contention because one or both takes
> longer
> > than its periodicity to complete so we eventually have more pending
> threads
> > than the cores can run, or...
>
>
> You could look at the imp of purge() on a Timer. It cause the entire queue
> to be rebuild and heapified again which is slow.
>
> Far far slower than synchronized. The synchronized blocks aren’t even
> really showing up in my timings so I’m not really worried about them. In
> fact removing them could introduce other issues so I”m hesitant to remove
> them without actual bugs.
>
What I wasn't understanding was that if we've enabled message expiration
checks, then when we GC a destination we have to cancel the message
expiration check timer (which we've chosen to do inefficiently by calling
purge()). I finally see the connection between those two things.
> > I'm in favor of the changes as you've
> > described it (I can't think of a reason we'd want advisory messages to
> > expire, though ideally it would be configurable for just in case someone
> > has a need for it that I can't think of; configurable or not, having 0 as
> > the default sounds like a clear win to me), I'm just not clear on how it
> > has the larger effects you describe.
> >
> >
> We can go either route now. My patches just refactored the way we remove
> task. Now what I do is I just use an AtomicReference and I can clear() it
> when I want to remove the timer. The underlying timer will get executed,
> but since it checks first that it’s non-null it,s just a quick noop once
> it’s cleared.
>
>
> > Also, can you point me to the O(N^2) code related to Timer.purge()? If
> it
> > really is purging every queue each time any queue's timer fires, that
> > sounds like incorrect code, but I'd like to see it to make sure I'm not
> > misunderstanding what you're describing.
> >
>
> I’m pretty sure it’s O(N^2)… what’s happening is that for each queue that
> needs to be GCd, it’s causing a purge. So if you have 1000 queues, it has
> to purge and rebuild it… then if you have 999 queues, and you remove one,
> it has to do a full purge and rebuild each time.
>
I hadn't expected purge() to call heapify(), so I was just expecting O(N)
runtime for the actual removal. The comments claim the runtime of the
purge operation is O(N + ClogN) where C is the number of deleted elements
(which would make it O(N) when C is 1, but looking at the code I think it's
actually O(N + ((N-C)/2)log((N-C)/2)), which is O(NlogN) when C is 1. So
it's not O(N^2), but it's also definitely not O(N) despite the comment's
claim.
Now I just have it essentially flag the timer task as clear and it will go
> through the timer queue and not get executed.
>
Sounds fine, as long as "flag the timer task as clear" also means "call
TimerTask.cancel(), so that the task's state will be TimerTask.CANCELLED so
it'll be removed the next time it comes to the top of the queue in
Timer.mainLoop(), in addition to any WeakReference stuff we may do". If
not, you're going to have a lot of no-op TimerTasks that stick around
forever, so 1) your value of N will grow over time, so even O(N) operations
will take longer over time, 2) there's still an expensive thread context
switch even if the task is a nominal no-op, and 3) eventually you could OOM
due to the memory those no-op TimerTasks take. Does that match what you've
implemented?
> If you’re still interested I have screenshots from my profiling that I can
> pull up.
>
> I might submit these with my PR to back up my claims.
>
> Kevin
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
Re: Setting expireMessagesPeriod=0 for advisory topics improved GC
performance by 2x..
Posted by Kevin Burton <bu...@spinn3r.com>.
> I wasn't at all clear why changing the frequency (to "never") of message
> expiration checks would affect the performance of destination GC operations
> when the number of queues is large. Is that due to synchronization between
> those two operations, or thread contention because one or both takes longer
> than its periodicity to complete so we eventually have more pending threads
> than the cores can run, or...
You could look at the imp of purge() on a Timer. It cause the entire queue
to be rebuild and heapified again which is slow.
Far far slower than synchronized. The synchronized blocks aren’t even
really showing up in my timings so I’m not really worried about them. In
fact removing them could introduce other issues so I”m hesitant to remove
them without actual bugs.
> I'm in favor of the changes as you've
> described it (I can't think of a reason we'd want advisory messages to
> expire, though ideally it would be configurable for just in case someone
> has a need for it that I can't think of; configurable or not, having 0 as
> the default sounds like a clear win to me), I'm just not clear on how it
> has the larger effects you describe.
>
>
We can go either route now. My patches just refactored the way we remove
task. Now what I do is I just use an AtomicReference and I can clear() it
when I want to remove the timer. The underlying timer will get executed,
but since it checks first that it’s non-null it,s just a quick noop once
it’s cleared.
> Also, can you point me to the O(N^2) code related to Timer.purge()? If it
> really is purging every queue each time any queue's timer fires, that
> sounds like incorrect code, but I'd like to see it to make sure I'm not
> misunderstanding what you're describing.
>
I’m pretty sure it’s O(N^2)… what’s happening is that for each queue that
needs to be GCd, it’s causing a purge. So if you have 1000 queues, it has
to purge and rebuild it… then if you have 999 queues, and you remove one,
it has to do a full purge and rebuild each time.
Now I just have it essentially flag the timer task as clear and it will go
through the timer queue and not get executed.
If you’re still interested I have screenshots from my profiling that I can
pull up.
I might submit these with my PR to back up my claims.
Kevin
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Setting expireMessagesPeriod=0 for advisory topics improved GC
performance by 2x..
Posted by Tim Bain <tb...@alumni.duke.edu>.
Kevin,
Messages don't pile up on advisory topics because of the semantics of
topics: unless you have a durable subscription (which the broker-to-broker
advisory subscriptions aren't, if I recall correctly), any message that
arrives on the broker will be delivered to all connected consumers (and not
to any disconnected ones), so the only thing that could cause messages to
back up is a slow consumer (which I've never observed to happen, though I'm
sure it's possible). You could certainly get slow consumers and/or durable
subscriptions on advisory topics if you've written custom consumers, just
like for any other topic, but that's a separate issue.
I wasn't at all clear why changing the frequency (to "never") of message
expiration checks would affect the performance of destination GC operations
when the number of queues is large. Is that due to synchronization between
those two operations, or thread contention because one or both takes longer
than its periodicity to complete so we eventually have more pending threads
than the cores can run, or... I'm in favor of the changes as you've
described it (I can't think of a reason we'd want advisory messages to
expire, though ideally it would be configurable for just in case someone
has a need for it that I can't think of; configurable or not, having 0 as
the default sounds like a clear win to me), I'm just not clear on how it
has the larger effects you describe.
Also, can you point me to the O(N^2) code related to Timer.purge()? If it
really is purging every queue each time any queue's timer fires, that
sounds like incorrect code, but I'd like to see it to make sure I'm not
misunderstanding what you're describing.
Tim
On May 2, 2015 4:34 PM, "Kevin Burton" <bu...@spinn3r.com> wrote:
> I’m confused by something. Why don’t messages pile up in advisory topics?
> Topics only deliver messages to consumers who are actually listening I
> assume?
>
> On Sat, May 2, 2015 at 2:07 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
> > I’m sorry.. this is N^2 because the Timer just keeps growing for every
> new
> > queue and they need to be purged and it has to evaluate each one.
> >
> > On Sat, May 2, 2015 at 1:40 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> >
> >> Also, while this is a small performance boost in my example, this should
> >> have a massive impact when running with large queues. It’s calling
> >> Timer.purge() on each queue GC and if you have a lot of queues you’re
> going
> >> to have a lot of TimerTasks so this is a O(N). It also holds a global
> lock
> >> during this operation so no new queues can be created during a queue GC.
> >>
> >> On Sat, May 2, 2015 at 12:23 PM, Kevin Burton <bu...@spinn3r.com>
> wrote:
> >>
> >>> I’m doing a bunch of performance analysis of ActiveMQ this weekend to
> >>> see if I can improve queue creation and destruction time. The good
> news is
> >>> that there are a lot of areas of optimization.
> >>>
> >>> It looks like one is that advisory topics are created with the default
> >>> expireMessagesPeriod (which is 30000). However, no advisory messages
> are
> >>> produced with TTLs.
> >>>
> >>> I believe this is done in AdvisoryBroker.fireAdvisory
> >>>
> >>> … looking at this code, a TTL isn’t set. So I think that it would be a
> >>> good idea to enable expireMessagesPeriod=0 for advisory topics.
> >>>
> >>> Here’s the code if you’re interested:
> >>>
> >>> http://pastebin.com/i3hXDGEW
> >>>
> >>> Looks like the following policyEntry works as a proof of concept.
> >>>
> >>> <policyEntry topic=">"
> >>> prioritizedMessages="false"
> >>> useCache="false"
> >>> maxBrowsePageSize="2147483647"
> >>> expireMessagesPeriod="0">
> >>>
> >>> </policyEntry>
> >>>
> >>> (I am going to constrain it for advisory topics though).
> >>>
> >>> In my test time time for GCing all my queues went from 45s to 17s.
> It’s
> >>> MUCH higher than this under production load though due threading
> issues.
> >>>
> >>> Are there any potential issues I might be missing here?
> >>>
> >>> If this works I might try to submit a patch to disable
> >>> expireMessagesPeriod by default for advisory topics.
> >>>
> >>> --
> >>>
> >>> Founder/CEO Spinn3r.com
> >>> Location: *San Francisco, CA*
> >>> blog: http://burtonator.wordpress.com
> >>> … or check out my Google+ profile
> >>> <https://plus.google.com/102718274791889610666/posts>
> >>>
> >>>
> >>
> >>
> >> --
> >>
> >> Founder/CEO Spinn3r.com
> >> Location: *San Francisco, CA*
> >> blog: http://burtonator.wordpress.com
> >> … or check out my Google+ profile
> >> <https://plus.google.com/102718274791889610666/posts>
> >>
> >>
> >
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> >
> >
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
Re: Setting expireMessagesPeriod=0 for advisory topics improved GC
performance by 2x..
Posted by Kevin Burton <bu...@spinn3r.com>.
I’m confused by something. Why don’t messages pile up in advisory topics?
Topics only deliver messages to consumers who are actually listening I
assume?
On Sat, May 2, 2015 at 2:07 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> I’m sorry.. this is N^2 because the Timer just keeps growing for every new
> queue and they need to be purged and it has to evaluate each one.
>
> On Sat, May 2, 2015 at 1:40 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> Also, while this is a small performance boost in my example, this should
>> have a massive impact when running with large queues. It’s calling
>> Timer.purge() on each queue GC and if you have a lot of queues you’re going
>> to have a lot of TimerTasks so this is a O(N). It also holds a global lock
>> during this operation so no new queues can be created during a queue GC.
>>
>> On Sat, May 2, 2015 at 12:23 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>>
>>> I’m doing a bunch of performance analysis of ActiveMQ this weekend to
>>> see if I can improve queue creation and destruction time. The good news is
>>> that there are a lot of areas of optimization.
>>>
>>> It looks like one is that advisory topics are created with the default
>>> expireMessagesPeriod (which is 30000). However, no advisory messages are
>>> produced with TTLs.
>>>
>>> I believe this is done in AdvisoryBroker.fireAdvisory
>>>
>>> … looking at this code, a TTL isn’t set. So I think that it would be a
>>> good idea to enable expireMessagesPeriod=0 for advisory topics.
>>>
>>> Here’s the code if you’re interested:
>>>
>>> http://pastebin.com/i3hXDGEW
>>>
>>> Looks like the following policyEntry works as a proof of concept.
>>>
>>> <policyEntry topic=">"
>>> prioritizedMessages="false"
>>> useCache="false"
>>> maxBrowsePageSize="2147483647"
>>> expireMessagesPeriod="0">
>>>
>>> </policyEntry>
>>>
>>> (I am going to constrain it for advisory topics though).
>>>
>>> In my test time time for GCing all my queues went from 45s to 17s. It’s
>>> MUCH higher than this under production load though due threading issues.
>>>
>>> Are there any potential issues I might be missing here?
>>>
>>> If this works I might try to submit a patch to disable
>>> expireMessagesPeriod by default for advisory topics.
>>>
>>> --
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> <https://plus.google.com/102718274791889610666/posts>
>>>
>>>
>>
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>>
>>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Setting expireMessagesPeriod=0 for advisory topics improved GC
performance by 2x..
Posted by Kevin Burton <bu...@spinn3r.com>.
I’m sorry.. this is N^2 because the Timer just keeps growing for every new
queue and they need to be purged and it has to evaluate each one.
On Sat, May 2, 2015 at 1:40 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> Also, while this is a small performance boost in my example, this should
> have a massive impact when running with large queues. It’s calling
> Timer.purge() on each queue GC and if you have a lot of queues you’re going
> to have a lot of TimerTasks so this is a O(N). It also holds a global lock
> during this operation so no new queues can be created during a queue GC.
>
> On Sat, May 2, 2015 at 12:23 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> I’m doing a bunch of performance analysis of ActiveMQ this weekend to see
>> if I can improve queue creation and destruction time. The good news is
>> that there are a lot of areas of optimization.
>>
>> It looks like one is that advisory topics are created with the default
>> expireMessagesPeriod (which is 30000). However, no advisory messages are
>> produced with TTLs.
>>
>> I believe this is done in AdvisoryBroker.fireAdvisory
>>
>> … looking at this code, a TTL isn’t set. So I think that it would be a
>> good idea to enable expireMessagesPeriod=0 for advisory topics.
>>
>> Here’s the code if you’re interested:
>>
>> http://pastebin.com/i3hXDGEW
>>
>> Looks like the following policyEntry works as a proof of concept.
>>
>> <policyEntry topic=">"
>> prioritizedMessages="false"
>> useCache="false"
>> maxBrowsePageSize="2147483647"
>> expireMessagesPeriod="0">
>>
>> </policyEntry>
>>
>> (I am going to constrain it for advisory topics though).
>>
>> In my test time time for GCing all my queues went from 45s to 17s. It’s
>> MUCH higher than this under production load though due threading issues.
>>
>> Are there any potential issues I might be missing here?
>>
>> If this works I might try to submit a patch to disable
>> expireMessagesPeriod by default for advisory topics.
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>>
>>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Setting expireMessagesPeriod=0 for advisory topics improved GC
performance by 2x..
Posted by Kevin Burton <bu...@spinn3r.com>.
Also, while this is a small performance boost in my example, this should
have a massive impact when running with large queues. It’s calling
Timer.purge() on each queue GC and if you have a lot of queues you’re going
to have a lot of TimerTasks so this is a O(N). It also holds a global lock
during this operation so no new queues can be created during a queue GC.
On Sat, May 2, 2015 at 12:23 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> I’m doing a bunch of performance analysis of ActiveMQ this weekend to see
> if I can improve queue creation and destruction time. The good news is
> that there are a lot of areas of optimization.
>
> It looks like one is that advisory topics are created with the default
> expireMessagesPeriod (which is 30000). However, no advisory messages are
> produced with TTLs.
>
> I believe this is done in AdvisoryBroker.fireAdvisory
>
> … looking at this code, a TTL isn’t set. So I think that it would be a
> good idea to enable expireMessagesPeriod=0 for advisory topics.
>
> Here’s the code if you’re interested:
>
> http://pastebin.com/i3hXDGEW
>
> Looks like the following policyEntry works as a proof of concept.
>
> <policyEntry topic=">"
> prioritizedMessages="false"
> useCache="false"
> maxBrowsePageSize="2147483647"
> expireMessagesPeriod="0">
>
> </policyEntry>
>
> (I am going to constrain it for advisory topics though).
>
> In my test time time for GCing all my queues went from 45s to 17s. It’s
> MUCH higher than this under production load though due threading issues.
>
> Are there any potential issues I might be missing here?
>
> If this works I might try to submit a patch to disable
> expireMessagesPeriod by default for advisory topics.
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>