You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Ilya Kasnacheev <il...@gmail.com> on 2018/03/05 11:46:10 UTC
Why does DataStreamer swallow exceptions?
Dear Igniters, why do I have a hunch that DataStreamer would readily
swallow exceptions?
DataStreamerImpl:1756 swallows marshalling error, lines 1774 & 1781 eat
deployment errors.
Some people are worried they can fill a leaking vessel without noticing
anything off.
Also in line 2156 fsync() on WAL can throw exceptions, which will be
swallowed, and IMP this fsync doesn't belong to DataStreamer code.
This question is not purely theoretical, we have also replaced one of these
eaters with throw with IGNITE-7519.
There was a fault in PDS implementation, which did not propagate to client.
A serious issue IMHO.
As a data streamer user I will expect that flush()/close() will throw any
pending exceptions and will only be silent if all data landed safely in the
cluster.
I also have this feeling that DataStreamer was written using very internal
APIs so that it can compromise guarantees that cache and SQL APIs are bound
by. I think I've heard something about not recovering properly in case of
node failures.
I think it's time to end this, if that was the case. DataStreamer should
not be a special case and it should guarantee data safety. WDYT?
--
Ilya Kasnacheev
Re: Why does DataStreamer swallow exceptions?
Posted by Ilya Kasnacheev <il...@gmail.com>.
I have created pull requests for both of tickets.
https://github.com/apache/ignite/pull/3651/files please comment if you have
strong reservations against this usability fix.
Note that it doesn't change timeouted get() behaviour because we can't know
in advance how long flush() will take.
--
Ilya Kasnacheev
2018-03-15 13:25 GMT+03:00 Ilya Kasnacheev <il...@gmail.com>:
> Hello!
>
> I have filed an issue https://issues.apache.org/jira/browse/IGNITE-7962 -
> going to work on it soon.
>
> I have also filed a ticket about usability issue:
> https://issues.apache.org/jira/browse/IGNITE-7963
> Feedback is appreciated.
>
> Regards,
>
> --
> Ilya Kasnacheev
>
> 2018-03-14 23:35 GMT+03:00 Valentin Kulichenko <
> valentin.kulichenko@gmail.com>:
>
>> Ilya,
>>
>> Yes, this is definitely a bug, please create a ticket.
>>
>> -Val
>>
>> On Wed, Mar 14, 2018 at 2:16 AM, Ilya Kasnacheev <
>> ilya.kasnacheev@gmail.com>
>> wrote:
>>
>> > Hello Val!
>> >
>> > No, this does NOT happen. If you collect future, call get() on it
>> later, it
>> > will finish normally despite exceptions in server log and entry not
>> being
>> > written. I will do some digging to figure out why this happens exactly.
>> >
>> > There's also another huge problems with DataStreamer's futures. They
>> never
>> > finish unless you call flush() on DS before calling get() on futures.
>> > I think this is a colossal usability problem (I'm pretty sure I've seen
>> > numerous messages about it on userlist) and I'll fill an issue if
>> nobody is
>> > objecting.
>> >
>> > Ilya.
>> >
>> > --
>> > Ilya Kasnacheev
>> >
>> > 2018-03-05 22:50 GMT+03:00 Valentin Kulichenko <
>> > valentin.kulichenko@gmail.com>:
>> >
>> > > Ilya,
>> > >
>> > > IgniteDataStreamer#addData method returns future which should be
>> > completed
>> > > with error if one is thrown on server side. Does this happen or not?
>> > >
>> > > -Val
>> > >
>> > > On Mon, Mar 5, 2018 at 4:10 AM, Nikolay Izhikov <ni...@apache.org>
>> > > wrote:
>> > >
>> > > > Hello, Ilya.
>> > > >
>> > > > > I think it's time to end this, if that was the case. DataStreamer
>> > > should
>> > > > > not be a special case and it should guarantee data safety. WDYT?
>> > > >
>> > > > +1 from me.
>> > > >
>> > > > I'm also facing this issue.
>> > > >
>> > > > Ticket - https://issues.apache.org/jira/browse/IGNITE-7756
>> > > > Discussion - http://apache-ignite-developers.2346864.n4.nabble.
>> > > > com/IgniteDataStreamer-silently-fails-on-a-server-node-td27239.html
>> > > >
>> > > >
>> > > >
>> > > > В Пн, 05/03/2018 в 14:46 +0300, Ilya Kasnacheev пишет:
>> > > > > Dear Igniters, why do I have a hunch that DataStreamer would
>> readily
>> > > > > swallow exceptions?
>> > > > >
>> > > > > DataStreamerImpl:1756 swallows marshalling error, lines 1774 &
>> 1781
>> > eat
>> > > > > deployment errors.
>> > > > >
>> > > > > Some people are worried they can fill a leaking vessel without
>> > noticing
>> > > > > anything off.
>> > > > > Also in line 2156 fsync() on WAL can throw exceptions, which will
>> be
>> > > > > swallowed, and IMP this fsync doesn't belong to DataStreamer code.
>> > > > >
>> > > > > This question is not purely theoretical, we have also replaced
>> one of
>> > > > these
>> > > > > eaters with throw with IGNITE-7519.
>> > > > >
>> > > > > There was a fault in PDS implementation, which did not propagate
>> to
>> > > > client.
>> > > > > A serious issue IMHO.
>> > > > >
>> > > > > As a data streamer user I will expect that flush()/close() will
>> throw
>> > > any
>> > > > > pending exceptions and will only be silent if all data landed
>> safely
>> > in
>> > > > the
>> > > > > cluster.
>> > > > >
>> > > > > I also have this feeling that DataStreamer was written using very
>> > > > internal
>> > > > > APIs so that it can compromise guarantees that cache and SQL APIs
>> are
>> > > > bound
>> > > > > by. I think I've heard something about not recovering properly in
>> > case
>> > > of
>> > > > > node failures.
>> > > > > I think it's time to end this, if that was the case. DataStreamer
>> > > should
>> > > > > not be a special case and it should guarantee data safety. WDYT?
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Why does DataStreamer swallow exceptions?
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I have filed an issue https://issues.apache.org/jira/browse/IGNITE-7962 -
going to work on it soon.
I have also filed a ticket about usability issue:
https://issues.apache.org/jira/browse/IGNITE-7963
Feedback is appreciated.
Regards,
--
Ilya Kasnacheev
2018-03-14 23:35 GMT+03:00 Valentin Kulichenko <
valentin.kulichenko@gmail.com>:
> Ilya,
>
> Yes, this is definitely a bug, please create a ticket.
>
> -Val
>
> On Wed, Mar 14, 2018 at 2:16 AM, Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com>
> wrote:
>
> > Hello Val!
> >
> > No, this does NOT happen. If you collect future, call get() on it later,
> it
> > will finish normally despite exceptions in server log and entry not being
> > written. I will do some digging to figure out why this happens exactly.
> >
> > There's also another huge problems with DataStreamer's futures. They
> never
> > finish unless you call flush() on DS before calling get() on futures.
> > I think this is a colossal usability problem (I'm pretty sure I've seen
> > numerous messages about it on userlist) and I'll fill an issue if nobody
> is
> > objecting.
> >
> > Ilya.
> >
> > --
> > Ilya Kasnacheev
> >
> > 2018-03-05 22:50 GMT+03:00 Valentin Kulichenko <
> > valentin.kulichenko@gmail.com>:
> >
> > > Ilya,
> > >
> > > IgniteDataStreamer#addData method returns future which should be
> > completed
> > > with error if one is thrown on server side. Does this happen or not?
> > >
> > > -Val
> > >
> > > On Mon, Mar 5, 2018 at 4:10 AM, Nikolay Izhikov <ni...@apache.org>
> > > wrote:
> > >
> > > > Hello, Ilya.
> > > >
> > > > > I think it's time to end this, if that was the case. DataStreamer
> > > should
> > > > > not be a special case and it should guarantee data safety. WDYT?
> > > >
> > > > +1 from me.
> > > >
> > > > I'm also facing this issue.
> > > >
> > > > Ticket - https://issues.apache.org/jira/browse/IGNITE-7756
> > > > Discussion - http://apache-ignite-developers.2346864.n4.nabble.
> > > > com/IgniteDataStreamer-silently-fails-on-a-server-node-td27239.html
> > > >
> > > >
> > > >
> > > > В Пн, 05/03/2018 в 14:46 +0300, Ilya Kasnacheev пишет:
> > > > > Dear Igniters, why do I have a hunch that DataStreamer would
> readily
> > > > > swallow exceptions?
> > > > >
> > > > > DataStreamerImpl:1756 swallows marshalling error, lines 1774 & 1781
> > eat
> > > > > deployment errors.
> > > > >
> > > > > Some people are worried they can fill a leaking vessel without
> > noticing
> > > > > anything off.
> > > > > Also in line 2156 fsync() on WAL can throw exceptions, which will
> be
> > > > > swallowed, and IMP this fsync doesn't belong to DataStreamer code.
> > > > >
> > > > > This question is not purely theoretical, we have also replaced one
> of
> > > > these
> > > > > eaters with throw with IGNITE-7519.
> > > > >
> > > > > There was a fault in PDS implementation, which did not propagate to
> > > > client.
> > > > > A serious issue IMHO.
> > > > >
> > > > > As a data streamer user I will expect that flush()/close() will
> throw
> > > any
> > > > > pending exceptions and will only be silent if all data landed
> safely
> > in
> > > > the
> > > > > cluster.
> > > > >
> > > > > I also have this feeling that DataStreamer was written using very
> > > > internal
> > > > > APIs so that it can compromise guarantees that cache and SQL APIs
> are
> > > > bound
> > > > > by. I think I've heard something about not recovering properly in
> > case
> > > of
> > > > > node failures.
> > > > > I think it's time to end this, if that was the case. DataStreamer
> > > should
> > > > > not be a special case and it should guarantee data safety. WDYT?
> > > > >
> > > >
> > >
> >
>
Re: Why does DataStreamer swallow exceptions?
Posted by Valentin Kulichenko <va...@gmail.com>.
Ilya,
Yes, this is definitely a bug, please create a ticket.
-Val
On Wed, Mar 14, 2018 at 2:16 AM, Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello Val!
>
> No, this does NOT happen. If you collect future, call get() on it later, it
> will finish normally despite exceptions in server log and entry not being
> written. I will do some digging to figure out why this happens exactly.
>
> There's also another huge problems with DataStreamer's futures. They never
> finish unless you call flush() on DS before calling get() on futures.
> I think this is a colossal usability problem (I'm pretty sure I've seen
> numerous messages about it on userlist) and I'll fill an issue if nobody is
> objecting.
>
> Ilya.
>
> --
> Ilya Kasnacheev
>
> 2018-03-05 22:50 GMT+03:00 Valentin Kulichenko <
> valentin.kulichenko@gmail.com>:
>
> > Ilya,
> >
> > IgniteDataStreamer#addData method returns future which should be
> completed
> > with error if one is thrown on server side. Does this happen or not?
> >
> > -Val
> >
> > On Mon, Mar 5, 2018 at 4:10 AM, Nikolay Izhikov <ni...@apache.org>
> > wrote:
> >
> > > Hello, Ilya.
> > >
> > > > I think it's time to end this, if that was the case. DataStreamer
> > should
> > > > not be a special case and it should guarantee data safety. WDYT?
> > >
> > > +1 from me.
> > >
> > > I'm also facing this issue.
> > >
> > > Ticket - https://issues.apache.org/jira/browse/IGNITE-7756
> > > Discussion - http://apache-ignite-developers.2346864.n4.nabble.
> > > com/IgniteDataStreamer-silently-fails-on-a-server-node-td27239.html
> > >
> > >
> > >
> > > В Пн, 05/03/2018 в 14:46 +0300, Ilya Kasnacheev пишет:
> > > > Dear Igniters, why do I have a hunch that DataStreamer would readily
> > > > swallow exceptions?
> > > >
> > > > DataStreamerImpl:1756 swallows marshalling error, lines 1774 & 1781
> eat
> > > > deployment errors.
> > > >
> > > > Some people are worried they can fill a leaking vessel without
> noticing
> > > > anything off.
> > > > Also in line 2156 fsync() on WAL can throw exceptions, which will be
> > > > swallowed, and IMP this fsync doesn't belong to DataStreamer code.
> > > >
> > > > This question is not purely theoretical, we have also replaced one of
> > > these
> > > > eaters with throw with IGNITE-7519.
> > > >
> > > > There was a fault in PDS implementation, which did not propagate to
> > > client.
> > > > A serious issue IMHO.
> > > >
> > > > As a data streamer user I will expect that flush()/close() will throw
> > any
> > > > pending exceptions and will only be silent if all data landed safely
> in
> > > the
> > > > cluster.
> > > >
> > > > I also have this feeling that DataStreamer was written using very
> > > internal
> > > > APIs so that it can compromise guarantees that cache and SQL APIs are
> > > bound
> > > > by. I think I've heard something about not recovering properly in
> case
> > of
> > > > node failures.
> > > > I think it's time to end this, if that was the case. DataStreamer
> > should
> > > > not be a special case and it should guarantee data safety. WDYT?
> > > >
> > >
> >
>
Re: Why does DataStreamer swallow exceptions?
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello Val!
No, this does NOT happen. If you collect future, call get() on it later, it
will finish normally despite exceptions in server log and entry not being
written. I will do some digging to figure out why this happens exactly.
There's also another huge problems with DataStreamer's futures. They never
finish unless you call flush() on DS before calling get() on futures.
I think this is a colossal usability problem (I'm pretty sure I've seen
numerous messages about it on userlist) and I'll fill an issue if nobody is
objecting.
Ilya.
--
Ilya Kasnacheev
2018-03-05 22:50 GMT+03:00 Valentin Kulichenko <
valentin.kulichenko@gmail.com>:
> Ilya,
>
> IgniteDataStreamer#addData method returns future which should be completed
> with error if one is thrown on server side. Does this happen or not?
>
> -Val
>
> On Mon, Mar 5, 2018 at 4:10 AM, Nikolay Izhikov <ni...@apache.org>
> wrote:
>
> > Hello, Ilya.
> >
> > > I think it's time to end this, if that was the case. DataStreamer
> should
> > > not be a special case and it should guarantee data safety. WDYT?
> >
> > +1 from me.
> >
> > I'm also facing this issue.
> >
> > Ticket - https://issues.apache.org/jira/browse/IGNITE-7756
> > Discussion - http://apache-ignite-developers.2346864.n4.nabble.
> > com/IgniteDataStreamer-silently-fails-on-a-server-node-td27239.html
> >
> >
> >
> > В Пн, 05/03/2018 в 14:46 +0300, Ilya Kasnacheev пишет:
> > > Dear Igniters, why do I have a hunch that DataStreamer would readily
> > > swallow exceptions?
> > >
> > > DataStreamerImpl:1756 swallows marshalling error, lines 1774 & 1781 eat
> > > deployment errors.
> > >
> > > Some people are worried they can fill a leaking vessel without noticing
> > > anything off.
> > > Also in line 2156 fsync() on WAL can throw exceptions, which will be
> > > swallowed, and IMP this fsync doesn't belong to DataStreamer code.
> > >
> > > This question is not purely theoretical, we have also replaced one of
> > these
> > > eaters with throw with IGNITE-7519.
> > >
> > > There was a fault in PDS implementation, which did not propagate to
> > client.
> > > A serious issue IMHO.
> > >
> > > As a data streamer user I will expect that flush()/close() will throw
> any
> > > pending exceptions and will only be silent if all data landed safely in
> > the
> > > cluster.
> > >
> > > I also have this feeling that DataStreamer was written using very
> > internal
> > > APIs so that it can compromise guarantees that cache and SQL APIs are
> > bound
> > > by. I think I've heard something about not recovering properly in case
> of
> > > node failures.
> > > I think it's time to end this, if that was the case. DataStreamer
> should
> > > not be a special case and it should guarantee data safety. WDYT?
> > >
> >
>
Re: Why does DataStreamer swallow exceptions?
Posted by Valentin Kulichenko <va...@gmail.com>.
Ilya,
IgniteDataStreamer#addData method returns future which should be completed
with error if one is thrown on server side. Does this happen or not?
-Val
On Mon, Mar 5, 2018 at 4:10 AM, Nikolay Izhikov <ni...@apache.org> wrote:
> Hello, Ilya.
>
> > I think it's time to end this, if that was the case. DataStreamer should
> > not be a special case and it should guarantee data safety. WDYT?
>
> +1 from me.
>
> I'm also facing this issue.
>
> Ticket - https://issues.apache.org/jira/browse/IGNITE-7756
> Discussion - http://apache-ignite-developers.2346864.n4.nabble.
> com/IgniteDataStreamer-silently-fails-on-a-server-node-td27239.html
>
>
>
> В Пн, 05/03/2018 в 14:46 +0300, Ilya Kasnacheev пишет:
> > Dear Igniters, why do I have a hunch that DataStreamer would readily
> > swallow exceptions?
> >
> > DataStreamerImpl:1756 swallows marshalling error, lines 1774 & 1781 eat
> > deployment errors.
> >
> > Some people are worried they can fill a leaking vessel without noticing
> > anything off.
> > Also in line 2156 fsync() on WAL can throw exceptions, which will be
> > swallowed, and IMP this fsync doesn't belong to DataStreamer code.
> >
> > This question is not purely theoretical, we have also replaced one of
> these
> > eaters with throw with IGNITE-7519.
> >
> > There was a fault in PDS implementation, which did not propagate to
> client.
> > A serious issue IMHO.
> >
> > As a data streamer user I will expect that flush()/close() will throw any
> > pending exceptions and will only be silent if all data landed safely in
> the
> > cluster.
> >
> > I also have this feeling that DataStreamer was written using very
> internal
> > APIs so that it can compromise guarantees that cache and SQL APIs are
> bound
> > by. I think I've heard something about not recovering properly in case of
> > node failures.
> > I think it's time to end this, if that was the case. DataStreamer should
> > not be a special case and it should guarantee data safety. WDYT?
> >
>
Re: Why does DataStreamer swallow exceptions?
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Ilya.
> I think it's time to end this, if that was the case. DataStreamer should
> not be a special case and it should guarantee data safety. WDYT?
+1 from me.
I'm also facing this issue.
Ticket - https://issues.apache.org/jira/browse/IGNITE-7756
Discussion - http://apache-ignite-developers.2346864.n4.nabble.com/IgniteDataStreamer-silently-fails-on-a-server-node-td27239.html
В Пн, 05/03/2018 в 14:46 +0300, Ilya Kasnacheev пишет:
> Dear Igniters, why do I have a hunch that DataStreamer would readily
> swallow exceptions?
>
> DataStreamerImpl:1756 swallows marshalling error, lines 1774 & 1781 eat
> deployment errors.
>
> Some people are worried they can fill a leaking vessel without noticing
> anything off.
> Also in line 2156 fsync() on WAL can throw exceptions, which will be
> swallowed, and IMP this fsync doesn't belong to DataStreamer code.
>
> This question is not purely theoretical, we have also replaced one of these
> eaters with throw with IGNITE-7519.
>
> There was a fault in PDS implementation, which did not propagate to client.
> A serious issue IMHO.
>
> As a data streamer user I will expect that flush()/close() will throw any
> pending exceptions and will only be silent if all data landed safely in the
> cluster.
>
> I also have this feeling that DataStreamer was written using very internal
> APIs so that it can compromise guarantees that cache and SQL APIs are bound
> by. I think I've heard something about not recovering properly in case of
> node failures.
> I think it's time to end this, if that was the case. DataStreamer should
> not be a special case and it should guarantee data safety. WDYT?
>