You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mathieu Fenniak <ma...@replicon.com> on 2017/03/17 17:27:47 UTC
Streams RocksDBException with no message?
Hey all,
So... what does it mean to have a RocksDBException with a message that just
has a single character? "e", "q", "]"... I've seen a few. Has anyone seen
this before?
Two example exceptions:
https://gist.github.com/mfenniak/c56beb6d5058e2b21df0309aea224f12
Kafka Streams 0.10.2.0. Both of these errors occurred during state store
initialization. I'm running a single Kafka Streams thread per server, this
occurred on two servers about a half-hour apart.
Mathieu
Re: Streams RocksDBException with no message?
Posted by Michael Noll <mi...@confluent.io>.
We're talking about `ulimit` (CLI tool) and the `nofile` limit (number of
open files), which you can access via `ulimit -n`.
Examples:
https://access.redhat.com/solutions/61334
https://stackoverflow.com/questions/21515463/how-to-increase-maximum-file-open-limit-ulimit-in-ubuntu
Depending on the operating system, the default setting is often pretty low
(e.g. 1024). Bump this up to sth higher, like 16k or 32k. Of course, an
even better approach is to monitor this metric on your servers/brokers, and
-- with this collected information -- bump the setting to a reasonable
value for your environment.
-Michael
On Sun, Mar 26, 2017 at 7:41 PM, Sachin Mittal <sj...@gmail.com> wrote:
> Hi,
> Could you please tell us what did you change for ulimit and how.
>
> We also are seem to be facing same issue.
>
> Thanks
> Sachin
>
>
> On Tue, Mar 21, 2017 at 9:22 PM, Mathieu Fenniak <
> mathieu.fenniak@replicon.com> wrote:
>
> > Thanks Guozhang.
> >
> > For my part, turns out I was hitting ulimit on my open file descriptors.
> > Phew, easy to fix... once I figured it out. :-)
> >
> > Mathieu
> >
> >
> > On Fri, Mar 17, 2017 at 4:14 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Hi Mathieu,
> > >
> > > We are aware of that since long time ago and I have been looking into
> > this
> > > issue, turns out to be a known issue in RocksDB:
> > >
> > > https://github.com/facebook/rocksdb/issues/1688
> > >
> > > And the corresponding fix (https://github.com/facebook/
> rocksdb/pull/1714
> > )
> > > has been merged in master but marked for
> > >
> > > - v5.1.4 <https://github.com/facebook/rocksdb/releases/tag/v5.1.4>
> > >
> > > only while the latest release is 5.1.2.
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Fri, Mar 17, 2017 at 10:27 AM, Mathieu Fenniak <
> > > mathieu.fenniak@replicon.com> wrote:
> > >
> > > > Hey all,
> > > >
> > > > So... what does it mean to have a RocksDBException with a message
> that
> > > just
> > > > has a single character? "e", "q", "]"... I've seen a few. Has
> anyone
> > > seen
> > > > this before?
> > > >
> > > > Two example exceptions:
> > > > https://gist.github.com/mfenniak/c56beb6d5058e2b21df0309aea224f12
> > > >
> > > > Kafka Streams 0.10.2.0. Both of these errors occurred during state
> > store
> > > > initialization. I'm running a single Kafka Streams thread per
> server,
> > > this
> > > > occurred on two servers about a half-hour apart.
> > > >
> > > > Mathieu
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
Re: Streams RocksDBException with no message?
Posted by Sachin Mittal <sj...@gmail.com>.
Hi,
Could you please tell us what did you change for ulimit and how.
We also are seem to be facing same issue.
Thanks
Sachin
On Tue, Mar 21, 2017 at 9:22 PM, Mathieu Fenniak <
mathieu.fenniak@replicon.com> wrote:
> Thanks Guozhang.
>
> For my part, turns out I was hitting ulimit on my open file descriptors.
> Phew, easy to fix... once I figured it out. :-)
>
> Mathieu
>
>
> On Fri, Mar 17, 2017 at 4:14 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hi Mathieu,
> >
> > We are aware of that since long time ago and I have been looking into
> this
> > issue, turns out to be a known issue in RocksDB:
> >
> > https://github.com/facebook/rocksdb/issues/1688
> >
> > And the corresponding fix (https://github.com/facebook/rocksdb/pull/1714
> )
> > has been merged in master but marked for
> >
> > - v5.1.4 <https://github.com/facebook/rocksdb/releases/tag/v5.1.4>
> >
> > only while the latest release is 5.1.2.
> >
> >
> > Guozhang
> >
> >
> > On Fri, Mar 17, 2017 at 10:27 AM, Mathieu Fenniak <
> > mathieu.fenniak@replicon.com> wrote:
> >
> > > Hey all,
> > >
> > > So... what does it mean to have a RocksDBException with a message that
> > just
> > > has a single character? "e", "q", "]"... I've seen a few. Has anyone
> > seen
> > > this before?
> > >
> > > Two example exceptions:
> > > https://gist.github.com/mfenniak/c56beb6d5058e2b21df0309aea224f12
> > >
> > > Kafka Streams 0.10.2.0. Both of these errors occurred during state
> store
> > > initialization. I'm running a single Kafka Streams thread per server,
> > this
> > > occurred on two servers about a half-hour apart.
> > >
> > > Mathieu
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
Re: Streams RocksDBException with no message?
Posted by Mathieu Fenniak <ma...@replicon.com>.
Thanks Guozhang.
For my part, turns out I was hitting ulimit on my open file descriptors.
Phew, easy to fix... once I figured it out. :-)
Mathieu
On Fri, Mar 17, 2017 at 4:14 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Mathieu,
>
> We are aware of that since long time ago and I have been looking into this
> issue, turns out to be a known issue in RocksDB:
>
> https://github.com/facebook/rocksdb/issues/1688
>
> And the corresponding fix (https://github.com/facebook/rocksdb/pull/1714)
> has been merged in master but marked for
>
> - v5.1.4 <https://github.com/facebook/rocksdb/releases/tag/v5.1.4>
>
> only while the latest release is 5.1.2.
>
>
> Guozhang
>
>
> On Fri, Mar 17, 2017 at 10:27 AM, Mathieu Fenniak <
> mathieu.fenniak@replicon.com> wrote:
>
> > Hey all,
> >
> > So... what does it mean to have a RocksDBException with a message that
> just
> > has a single character? "e", "q", "]"... I've seen a few. Has anyone
> seen
> > this before?
> >
> > Two example exceptions:
> > https://gist.github.com/mfenniak/c56beb6d5058e2b21df0309aea224f12
> >
> > Kafka Streams 0.10.2.0. Both of these errors occurred during state store
> > initialization. I'm running a single Kafka Streams thread per server,
> this
> > occurred on two servers about a half-hour apart.
> >
> > Mathieu
> >
>
>
>
> --
> -- Guozhang
>
Re: Streams RocksDBException with no message?
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Mathieu,
We are aware of that since long time ago and I have been looking into this
issue, turns out to be a known issue in RocksDB:
https://github.com/facebook/rocksdb/issues/1688
And the corresponding fix (https://github.com/facebook/rocksdb/pull/1714)
has been merged in master but marked for
- v5.1.4 <https://github.com/facebook/rocksdb/releases/tag/v5.1.4>
only while the latest release is 5.1.2.
Guozhang
On Fri, Mar 17, 2017 at 10:27 AM, Mathieu Fenniak <
mathieu.fenniak@replicon.com> wrote:
> Hey all,
>
> So... what does it mean to have a RocksDBException with a message that just
> has a single character? "e", "q", "]"... I've seen a few. Has anyone seen
> this before?
>
> Two example exceptions:
> https://gist.github.com/mfenniak/c56beb6d5058e2b21df0309aea224f12
>
> Kafka Streams 0.10.2.0. Both of these errors occurred during state store
> initialization. I'm running a single Kafka Streams thread per server, this
> occurred on two servers about a half-hour apart.
>
> Mathieu
>
--
-- Guozhang