Service unavailable due to exceeding transaction memory consumption

[Nabeelah Harris <na...@impact.com>]

Hi there

While writing to a particular partition of a Kudu table using
KuduContext.insertRows, I receive the following error: "Service
unavailable: Transaction failed, tablet <tabletId> transaction memory
consumption (67075227) has exceeded its limit (67108864) or the limit
of an ancestral tracker". How would this occur is the consumption is
less than the limit? Writing to other partitions seems to work just
fine.

I am inclined to believe that this isn't due to an issue with the
tablet or tablet server itself, as I get the same error when I drop
and re-create the partition, and the partition ends up on a different
tablet/server entirely. Is this simply because I'm trying to write too
much data at once? What are the kinds of logs I can look for on the
masters/tablet servers to indicate that this might happen? What are
some of the configs I might be able to tweak regarding this issue?

Thanks
Nabeelah

Re: Service unavailable due to exceeding transaction memory consumption

[Adar Lieber-Dembo <ad...@cloudera.com>]

The error indicates that the set of outstanding transactions on this
tablet is hovering around 64MB. The error message is inherently racy
in that the values included aren't necessarily the values that
triggered the failure, so don't worry about that aspect of it.

A couple things to think about:
1. Normally the Kudu client retries a write if it receives a "service
unavailable" error. Does the Spark error indicate that retrying took
place? Was there an eventual timeout?
2. It's pretty unusual to be up against this 64M limit; it's usually a
sign that your write batches may be too large, or that the tablet
leader isn't able to replicate to its followers. What's this workload
look like? What is being inserted? How large are the rows? How many
rows per batch? What are your non-default configs? Does 'kudu cluster
ksck' come back healthy?
3. If you want to know more about the transaction state of a tserver
at any given time, visit the tserver's /transactions page in its web
UI (linked via "Dashboards").

On Fri, Feb 22, 2019 at 5:34 AM Nabeelah Harris
<na...@impact.com> wrote:
>
> Hi there
>
> While writing to a particular partition of a Kudu table using
> KuduContext.insertRows, I receive the following error: "Service
> unavailable: Transaction failed, tablet <tabletId> transaction memory
> consumption (67075227) has exceeded its limit (67108864) or the limit
> of an ancestral tracker". How would this occur is the consumption is
> less than the limit? Writing to other partitions seems to work just
> fine.
>
> I am inclined to believe that this isn't due to an issue with the
> tablet or tablet server itself, as I get the same error when I drop
> and re-create the partition, and the partition ends up on a different
> tablet/server entirely. Is this simply because I'm trying to write too
> much data at once? What are the kinds of logs I can look for on the
> masters/tablet servers to indicate that this might happen? What are
> some of the configs I might be able to tweak regarding this issue?
>
> Thanks
> Nabeelah