You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Ken Krugler <kk...@transpac.com> on 2018/05/04 02:56:07 UTC
Confusing error message in MemCheckpointStreamFactory with large
state size?
Currently in the MemCheckpointStreamFactory.checkSize() method, it can throw an IOException with:
throw new IOException(
"Size of the state is larger than the maximum permitted memory-backed state. Size="
+ size + " , maxSize=" + maxSize
+ " . Consider using a different state backend, like the File System State backend.");
But this will happen even if you’re using the File System State backend, right?
This came up here: https://stackoverflow.com/questions/50149005/ioexception-size-of-the-state-is-larger-than-the-maximum-permitted-memory-backe <https://stackoverflow.com/questions/50149005/ioexception-size-of-the-state-is-larger-than-the-maximum-permitted-memory-backe>
So shouldn’t it suggest using the RocksDB state backend instead?
— Ken
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra
Re: Confusing error message in MemCheckpointStreamFactory with large
state size?
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Ken and Rong,
That's a good point.
The FSStateBackend holds all state in the JVM heap as well. The name is
also not well chosen, IMO.
Thanks for creating the JIRA ticket!
Fabian
2018-05-05 0:52 GMT+02:00 Ken Krugler <kk...@transpac.com>:
>
> > On May 3, 2018, at 9:06 PM, Rong Rong <wa...@gmail.com> wrote:
> >
> > Agree it is definitely not intuitive trying to figure out what to do
> based
> > on this message.
> > I think the message should be changed to "please consider increasing
> > maximum permitted memory size, increase task manager parallelism, or
> using
> > a non-memory-based state backend".
> >
> > Could you please open a JIRA ticket in:
> > https://issues.apache.org/jira/projects/FLINK/issues/
>
> Sure, see
>
> https://issues.apache.org/jira/browse/FLINK-9300 <
> https://issues.apache.org/jira/browse/FLINK-9300>
>
> — Ken
>
> > On Thu, May 3, 2018 at 7:56 PM, Ken Krugler <kkrugler_lists@transpac.com
> >
> > wrote:
> >
> >> Currently in the MemCheckpointStreamFactory.checkSize() method, it can
> >> throw an IOException with:
> >>
> >> throw new IOException(
> >> "Size of the state is larger than
> >> the maximum permitted memory-backed state. Size="
> >> + size + " ,
> >> maxSize=" + maxSize
> >> + " . Consider
> >> using a different state backend, like the File System State backend.");
> >>
> >> But this will happen even if you’re using the File System State backend,
> >> right?
> >>
> >> This came up here: https://stackoverflow.com/questions/50149005/
> >> ioexception-size-of-the-state-is-larger-than-the-maximum-
> >> permitted-memory-backe <https://stackoverflow.com/questions/50149005/
> >> ioexception-size-of-the-state-is-larger-than-the-maximum-
> >> permitted-memory-backe>
> >>
> >> So shouldn’t it suggest using the RocksDB state backend instead?
> >>
> >> — Ken
>
> --------------------------------------------
> http://about.me/kkrugler
> +1 530-210-6378
>
>
Re: Confusing error message in MemCheckpointStreamFactory with large
state size?
Posted by Ken Krugler <kk...@transpac.com>.
> On May 3, 2018, at 9:06 PM, Rong Rong <wa...@gmail.com> wrote:
>
> Agree it is definitely not intuitive trying to figure out what to do based
> on this message.
> I think the message should be changed to "please consider increasing
> maximum permitted memory size, increase task manager parallelism, or using
> a non-memory-based state backend".
>
> Could you please open a JIRA ticket in:
> https://issues.apache.org/jira/projects/FLINK/issues/
Sure, see
https://issues.apache.org/jira/browse/FLINK-9300 <https://issues.apache.org/jira/browse/FLINK-9300>
— Ken
> On Thu, May 3, 2018 at 7:56 PM, Ken Krugler <kk...@transpac.com>
> wrote:
>
>> Currently in the MemCheckpointStreamFactory.checkSize() method, it can
>> throw an IOException with:
>>
>> throw new IOException(
>> "Size of the state is larger than
>> the maximum permitted memory-backed state. Size="
>> + size + " ,
>> maxSize=" + maxSize
>> + " . Consider
>> using a different state backend, like the File System State backend.");
>>
>> But this will happen even if you’re using the File System State backend,
>> right?
>>
>> This came up here: https://stackoverflow.com/questions/50149005/
>> ioexception-size-of-the-state-is-larger-than-the-maximum-
>> permitted-memory-backe <https://stackoverflow.com/questions/50149005/
>> ioexception-size-of-the-state-is-larger-than-the-maximum-
>> permitted-memory-backe>
>>
>> So shouldn’t it suggest using the RocksDB state backend instead?
>>
>> — Ken
--------------------------------------------
http://about.me/kkrugler
+1 530-210-6378
Re: Confusing error message in MemCheckpointStreamFactory with large
state size?
Posted by Rong Rong <wa...@gmail.com>.
Agree it is definitely not intuitive trying to figure out what to do based
on this message.
I think the message should be changed to "please consider increasing
maximum permitted memory size, increase task manager parallelism, or using
a non-memory-based state backend".
Could you please open a JIRA ticket in:
https://issues.apache.org/jira/projects/FLINK/issues/
Thanks,
Rong
On Thu, May 3, 2018 at 7:56 PM, Ken Krugler <kk...@transpac.com>
wrote:
> Currently in the MemCheckpointStreamFactory.checkSize() method, it can
> throw an IOException with:
>
> throw new IOException(
> "Size of the state is larger than
> the maximum permitted memory-backed state. Size="
> + size + " ,
> maxSize=" + maxSize
> + " . Consider
> using a different state backend, like the File System State backend.");
>
> But this will happen even if you’re using the File System State backend,
> right?
>
> This came up here: https://stackoverflow.com/questions/50149005/
> ioexception-size-of-the-state-is-larger-than-the-maximum-
> permitted-memory-backe <https://stackoverflow.com/questions/50149005/
> ioexception-size-of-the-state-is-larger-than-the-maximum-
> permitted-memory-backe>
>
> So shouldn’t it suggest using the RocksDB state backend instead?
>
> — Ken
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassandra
>
>