You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Logan Jones <lo...@codescratch.com> on 2022/11/04 19:40:16 UTC
Rollback to 1.9.3
Hello:
We recently upgraded from Accumulo 1.9.3 to 1.10.2. Is it safe to roll back
to Accumulo 1.9.3?
Thanks in advance,
- Logan
Re: Rollback to 1.9.3
Posted by Christopher <ct...@apache.org>.
I don't know what could go wrong, so it's hard to say that it would be
obvious.
For what it's worth, if we can identify a performance bug, we can release a
fix in a 1.10.3, so you can upgrade instead of downgrade.
On Fri, Nov 4, 2022, 16:26 Logan Jones <lo...@codescratch.com> wrote:
> Thanks all,
>
> We have a test system that we could try rolling back. If something does
> break, will it be obvious?
>
> Dave, the ingest rates are slightly more spikey, but I think it's mostly
> because tservers are bouncing and the cluster is working to catch up.
> Nothing major jumps out as an increase in throughput (i.e. ingest rate in
> terms of operations per second seem to be roughly equivalent. The same is
> true for the ingest rate in MB/s.)
>
> On Fri, Nov 4, 2022 at 4:21 PM Christopher <ct...@apache.org> wrote:
>
> > I don't think it has any changes that would prevent rollback, but it's
> not
> > a scenario that has been tested to my knowledge.
> >
> > On Fri, Nov 4, 2022, 16:15 Dave Marion <dm...@gmail.com> wrote:
> >
> > > It's going to take some time to review the changes[1], but I don't see
> > > changes in the default JVM sizes. I was wondering if maybe the issue is
> > > that it's running faster. You are loading the same amount of data, but
> is
> > > it going faster by chance? If so, you could be creating more garbage
> per
> > > unit time putting more pressure on the GC. Just a thought.
> > >
> > > [1] https://github.com/apache/accumulo/compare/rel/1.9.3..rel/1.10.2
> > >
> > > On Fri, Nov 4, 2022 at 4:02 PM Logan Jones <lo...@codescratch.com>
> > wrote:
> > >
> > > > Yeah, our memory usage is drastically different since the upgrade.
> > > >
> > > > We are seeing spikes in heap utilization on tablet servers that
> weren't
> > > > happening before the upgrade despite our ingest load being roughly
> the
> > > > same. This increase in heap utilization seems to be causing long GC
> > > times.
> > > > Those GC times are long enough that the tablet servers lose their
> locks
> > > and
> > > > then die.
> > > >
> > > > Looking into the JVM options, we don't see anything obvious that
> > changed
> > > > around the garbage collector, and looking at the Accumulo release
> notes
> > > > didn't leave us any indication that something like this should have
> > > > changed, but nevertheless we are seeing crashes of tservers. I'm
> mostly
> > > > trying to identify whether or not rollback is even an option.
> > > >
> > > > - Logan
> > > >
> > > > On Fri, Nov 4, 2022 at 3:49 PM Dave Marion <dm...@gmail.com>
> > wrote:
> > > >
> > > > > Are you running into an error or some other issue that is making
> > you
> > > > > think that you have to rollback? I don't know that rolling back has
> > > been
> > > > > tested.
> > > > >
> > > > > On Fri, Nov 4, 2022 at 3:40 PM Logan Jones <lo...@codescratch.com>
> > > > wrote:
> > > > >
> > > > > > Hello:
> > > > > >
> > > > > > We recently upgraded from Accumulo 1.9.3 to 1.10.2. Is it safe to
> > > roll
> > > > > back
> > > > > > to Accumulo 1.9.3?
> > > > > >
> > > > > > Thanks in advance,
> > > > > >
> > > > > > - Logan
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Rollback to 1.9.3
Posted by Logan Jones <lo...@codescratch.com>.
Thanks all,
We have a test system that we could try rolling back. If something does
break, will it be obvious?
Dave, the ingest rates are slightly more spikey, but I think it's mostly
because tservers are bouncing and the cluster is working to catch up.
Nothing major jumps out as an increase in throughput (i.e. ingest rate in
terms of operations per second seem to be roughly equivalent. The same is
true for the ingest rate in MB/s.)
On Fri, Nov 4, 2022 at 4:21 PM Christopher <ct...@apache.org> wrote:
> I don't think it has any changes that would prevent rollback, but it's not
> a scenario that has been tested to my knowledge.
>
> On Fri, Nov 4, 2022, 16:15 Dave Marion <dm...@gmail.com> wrote:
>
> > It's going to take some time to review the changes[1], but I don't see
> > changes in the default JVM sizes. I was wondering if maybe the issue is
> > that it's running faster. You are loading the same amount of data, but is
> > it going faster by chance? If so, you could be creating more garbage per
> > unit time putting more pressure on the GC. Just a thought.
> >
> > [1] https://github.com/apache/accumulo/compare/rel/1.9.3..rel/1.10.2
> >
> > On Fri, Nov 4, 2022 at 4:02 PM Logan Jones <lo...@codescratch.com>
> wrote:
> >
> > > Yeah, our memory usage is drastically different since the upgrade.
> > >
> > > We are seeing spikes in heap utilization on tablet servers that weren't
> > > happening before the upgrade despite our ingest load being roughly the
> > > same. This increase in heap utilization seems to be causing long GC
> > times.
> > > Those GC times are long enough that the tablet servers lose their locks
> > and
> > > then die.
> > >
> > > Looking into the JVM options, we don't see anything obvious that
> changed
> > > around the garbage collector, and looking at the Accumulo release notes
> > > didn't leave us any indication that something like this should have
> > > changed, but nevertheless we are seeing crashes of tservers. I'm mostly
> > > trying to identify whether or not rollback is even an option.
> > >
> > > - Logan
> > >
> > > On Fri, Nov 4, 2022 at 3:49 PM Dave Marion <dm...@gmail.com>
> wrote:
> > >
> > > > Are you running into an error or some other issue that is making
> you
> > > > think that you have to rollback? I don't know that rolling back has
> > been
> > > > tested.
> > > >
> > > > On Fri, Nov 4, 2022 at 3:40 PM Logan Jones <lo...@codescratch.com>
> > > wrote:
> > > >
> > > > > Hello:
> > > > >
> > > > > We recently upgraded from Accumulo 1.9.3 to 1.10.2. Is it safe to
> > roll
> > > > back
> > > > > to Accumulo 1.9.3?
> > > > >
> > > > > Thanks in advance,
> > > > >
> > > > > - Logan
> > > > >
> > > >
> > >
> >
>
Re: Rollback to 1.9.3
Posted by Christopher <ct...@apache.org>.
I don't think it has any changes that would prevent rollback, but it's not
a scenario that has been tested to my knowledge.
On Fri, Nov 4, 2022, 16:15 Dave Marion <dm...@gmail.com> wrote:
> It's going to take some time to review the changes[1], but I don't see
> changes in the default JVM sizes. I was wondering if maybe the issue is
> that it's running faster. You are loading the same amount of data, but is
> it going faster by chance? If so, you could be creating more garbage per
> unit time putting more pressure on the GC. Just a thought.
>
> [1] https://github.com/apache/accumulo/compare/rel/1.9.3..rel/1.10.2
>
> On Fri, Nov 4, 2022 at 4:02 PM Logan Jones <lo...@codescratch.com> wrote:
>
> > Yeah, our memory usage is drastically different since the upgrade.
> >
> > We are seeing spikes in heap utilization on tablet servers that weren't
> > happening before the upgrade despite our ingest load being roughly the
> > same. This increase in heap utilization seems to be causing long GC
> times.
> > Those GC times are long enough that the tablet servers lose their locks
> and
> > then die.
> >
> > Looking into the JVM options, we don't see anything obvious that changed
> > around the garbage collector, and looking at the Accumulo release notes
> > didn't leave us any indication that something like this should have
> > changed, but nevertheless we are seeing crashes of tservers. I'm mostly
> > trying to identify whether or not rollback is even an option.
> >
> > - Logan
> >
> > On Fri, Nov 4, 2022 at 3:49 PM Dave Marion <dm...@gmail.com> wrote:
> >
> > > Are you running into an error or some other issue that is making you
> > > think that you have to rollback? I don't know that rolling back has
> been
> > > tested.
> > >
> > > On Fri, Nov 4, 2022 at 3:40 PM Logan Jones <lo...@codescratch.com>
> > wrote:
> > >
> > > > Hello:
> > > >
> > > > We recently upgraded from Accumulo 1.9.3 to 1.10.2. Is it safe to
> roll
> > > back
> > > > to Accumulo 1.9.3?
> > > >
> > > > Thanks in advance,
> > > >
> > > > - Logan
> > > >
> > >
> >
>
Re: Rollback to 1.9.3
Posted by Dave Marion <dm...@gmail.com>.
It's going to take some time to review the changes[1], but I don't see
changes in the default JVM sizes. I was wondering if maybe the issue is
that it's running faster. You are loading the same amount of data, but is
it going faster by chance? If so, you could be creating more garbage per
unit time putting more pressure on the GC. Just a thought.
[1] https://github.com/apache/accumulo/compare/rel/1.9.3..rel/1.10.2
On Fri, Nov 4, 2022 at 4:02 PM Logan Jones <lo...@codescratch.com> wrote:
> Yeah, our memory usage is drastically different since the upgrade.
>
> We are seeing spikes in heap utilization on tablet servers that weren't
> happening before the upgrade despite our ingest load being roughly the
> same. This increase in heap utilization seems to be causing long GC times.
> Those GC times are long enough that the tablet servers lose their locks and
> then die.
>
> Looking into the JVM options, we don't see anything obvious that changed
> around the garbage collector, and looking at the Accumulo release notes
> didn't leave us any indication that something like this should have
> changed, but nevertheless we are seeing crashes of tservers. I'm mostly
> trying to identify whether or not rollback is even an option.
>
> - Logan
>
> On Fri, Nov 4, 2022 at 3:49 PM Dave Marion <dm...@gmail.com> wrote:
>
> > Are you running into an error or some other issue that is making you
> > think that you have to rollback? I don't know that rolling back has been
> > tested.
> >
> > On Fri, Nov 4, 2022 at 3:40 PM Logan Jones <lo...@codescratch.com>
> wrote:
> >
> > > Hello:
> > >
> > > We recently upgraded from Accumulo 1.9.3 to 1.10.2. Is it safe to roll
> > back
> > > to Accumulo 1.9.3?
> > >
> > > Thanks in advance,
> > >
> > > - Logan
> > >
> >
>
Re: Rollback to 1.9.3
Posted by Logan Jones <lo...@codescratch.com>.
Yeah, our memory usage is drastically different since the upgrade.
We are seeing spikes in heap utilization on tablet servers that weren't
happening before the upgrade despite our ingest load being roughly the
same. This increase in heap utilization seems to be causing long GC times.
Those GC times are long enough that the tablet servers lose their locks and
then die.
Looking into the JVM options, we don't see anything obvious that changed
around the garbage collector, and looking at the Accumulo release notes
didn't leave us any indication that something like this should have
changed, but nevertheless we are seeing crashes of tservers. I'm mostly
trying to identify whether or not rollback is even an option.
- Logan
On Fri, Nov 4, 2022 at 3:49 PM Dave Marion <dm...@gmail.com> wrote:
> Are you running into an error or some other issue that is making you
> think that you have to rollback? I don't know that rolling back has been
> tested.
>
> On Fri, Nov 4, 2022 at 3:40 PM Logan Jones <lo...@codescratch.com> wrote:
>
> > Hello:
> >
> > We recently upgraded from Accumulo 1.9.3 to 1.10.2. Is it safe to roll
> back
> > to Accumulo 1.9.3?
> >
> > Thanks in advance,
> >
> > - Logan
> >
>
Re: Rollback to 1.9.3
Posted by Dave Marion <dm...@gmail.com>.
Are you running into an error or some other issue that is making you
think that you have to rollback? I don't know that rolling back has been
tested.
On Fri, Nov 4, 2022 at 3:40 PM Logan Jones <lo...@codescratch.com> wrote:
> Hello:
>
> We recently upgraded from Accumulo 1.9.3 to 1.10.2. Is it safe to roll back
> to Accumulo 1.9.3?
>
> Thanks in advance,
>
> - Logan
>