You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by Leonard Lausen <le...@lausen.nl> on 2018/07/20 22:19:46 UTC
How should MXNet treat nan values?
Hello MXNet community,
It seems that there is currently no agreed upon principle to handle
`nan` values in operators. This has led to inconsistencies between
operators and also to inconsistency over releases. Some operators ignore
nan values (eg. argmax), others treated it as maximum (e.g. topk up to
mxnet v1.2) or just return “undefined” output (e.g. topk starting with
mxnet v1.3).
Initially the change in topk was reported as a bug
(https://github.com/apache/incubator-mxnet/issues/8510) as some users
relied on the behavior. However (and rightfully) @asmushetzel, who
contributed the improved topk operator for mxnet v1.3 pointed out that
the change did not break any documented behavior.
To go forward, please share your opinion how MXNet should handle `nan`
values. Should we continue to treat the behavior as undefined and
possibly silently changing between releases? Should we define a
reasonable standard (e.g. follow numpy) and treat operators that deviate
as buggy? Should we just document how operators behave currently and
warn if the behavior changes? Something else?
Please make your opinion known so above issue can be resolved/closed and
general guidelines can be defined for future contributions, following
whatever consensus emerges.
Thanks!
Leonard
Re: How should MXNet treat nan values?
Posted by Junru Shao <ju...@gmail.com>.
I think it is worth discussing.
NunPy has defined its own rules to handle with numeral exceptions, which makes a lot of sense to me. [Link](https://docs.scipy.org/doc/numpy/user/misc.html#how-numpy-handles-numerical-exceptions)
On 2018/07/20 22:19:46, Leonard Lausen <le...@lausen.nl> wrote:
> Hello MXNet community,
>
> It seems that there is currently no agreed upon principle to handle
> `nan` values in operators. This has led to inconsistencies between
> operators and also to inconsistency over releases. Some operators ignore
> nan values (eg. argmax), others treated it as maximum (e.g. topk up to
> mxnet v1.2) or just return “undefined” output (e.g. topk starting with
> mxnet v1.3).
>
> Initially the change in topk was reported as a bug
> (https://github.com/apache/incubator-mxnet/issues/8510) as some users
> relied on the behavior. However (and rightfully) @asmushetzel, who
> contributed the improved topk operator for mxnet v1.3 pointed out that
> the change did not break any documented behavior.
>
> To go forward, please share your opinion how MXNet should handle `nan`
> values. Should we continue to treat the behavior as undefined and
> possibly silently changing between releases? Should we define a
> reasonable standard (e.g. follow numpy) and treat operators that deviate
> as buggy? Should we just document how operators behave currently and
> warn if the behavior changes? Something else?
>
> Please make your opinion known so above issue can be resolved/closed and
> general guidelines can be defined for future contributions, following
> whatever consensus emerges.
>
> Thanks!
> Leonard
>
Re: How should MXNet treat nan values?
Posted by Chris Olivier <cj...@gmail.com>.
If you behave like numpy for sparse, then things like dividing any sparse
matrix by another sparse matrix will produce a dense matrix with a lot of
NaNs in it wherever it encountered a “missing” value in both the source and
destination positions of the sparse matrices (ie 0 divided by 0). If i
remember correctly, this was the process at first (creating a dense matrix
with lots of NaNs, which was equivalent to the same calculation with dense
matrices in numpy) but then was replaced by just assuming that you didn’t
want those NaNs and what you really wanted was a sparse result.
On Sat, Jul 21, 2018 at 12:31 PM Junru Shao <ju...@gmail.com> wrote:
> However, I am not 100% sure how much performance will be sacrificed if we
> stick to NumPy's approach which seems to check numeral exceptions on each
> step.
>
> I believe it will be great if we could make the default setting to be "no
> checking", and leave users an option to turn on these numeral exception
> checkings.
>
> On 2018/07/20 22:19:46, Leonard Lausen <le...@lausen.nl> wrote:
> > Hello MXNet community,
> >
> > It seems that there is currently no agreed upon principle to handle
> > `nan` values in operators. This has led to inconsistencies between
> > operators and also to inconsistency over releases. Some operators ignore
> > nan values (eg. argmax), others treated it as maximum (e.g. topk up to
> > mxnet v1.2) or just return “undefined” output (e.g. topk starting with
> > mxnet v1.3).
> >
> > Initially the change in topk was reported as a bug
> > (https://github.com/apache/incubator-mxnet/issues/8510) as some users
> > relied on the behavior. However (and rightfully) @asmushetzel, who
> > contributed the improved topk operator for mxnet v1.3 pointed out that
> > the change did not break any documented behavior.
> >
> > To go forward, please share your opinion how MXNet should handle `nan`
> > values. Should we continue to treat the behavior as undefined and
> > possibly silently changing between releases? Should we define a
> > reasonable standard (e.g. follow numpy) and treat operators that deviate
> > as buggy? Should we just document how operators behave currently and
> > warn if the behavior changes? Something else?
> >
> > Please make your opinion known so above issue can be resolved/closed and
> > general guidelines can be defined for future contributions, following
> > whatever consensus emerges.
> >
> > Thanks!
> > Leonard
> >
>
Re: How should MXNet treat nan values?
Posted by Asmus Hetzel <as...@yahoo.de.INVALID>.
+1
Performance should not be affected by default.
Generally, I would treat nan as undefined behaviour and allow to explicitly enable checks for nan which will throw exceptions.
Asmus
Am Samstag, 21. Juli 2018, 21:31:05 MESZ hat Junru Shao <ju...@gmail.com> Folgendes geschrieben:
However, I am not 100% sure how much performance will be sacrificed if we stick to NumPy's approach which seems to check numeral exceptions on each step.
I believe it will be great if we could make the default setting to be "no checking", and leave users an option to turn on these numeral exception checkings.
On 2018/07/20 22:19:46, Leonard Lausen <le...@lausen.nl> wrote:
> Hello MXNet community,
>
> It seems that there is currently no agreed upon principle to handle
> `nan` values in operators. This has led to inconsistencies between
> operators and also to inconsistency over releases. Some operators ignore
> nan values (eg. argmax), others treated it as maximum (e.g. topk up to
> mxnet v1.2) or just return “undefined” output (e.g. topk starting with
> mxnet v1.3).
>
> Initially the change in topk was reported as a bug
> (https://github.com/apache/incubator-mxnet/issues/8510) as some users
> relied on the behavior. However (and rightfully) @asmushetzel, who
> contributed the improved topk operator for mxnet v1.3 pointed out that
> the change did not break any documented behavior.
>
> To go forward, please share your opinion how MXNet should handle `nan`
> values. Should we continue to treat the behavior as undefined and
> possibly silently changing between releases? Should we define a
> reasonable standard (e.g. follow numpy) and treat operators that deviate
> as buggy? Should we just document how operators behave currently and
> warn if the behavior changes? Something else?
>
> Please make your opinion known so above issue can be resolved/closed and
> general guidelines can be defined for future contributions, following
> whatever consensus emerges.
>
> Thanks!
> Leonard
>
Re: How should MXNet treat nan values?
Posted by Junru Shao <ju...@gmail.com>.
However, I am not 100% sure how much performance will be sacrificed if we stick to NumPy's approach which seems to check numeral exceptions on each step.
I believe it will be great if we could make the default setting to be "no checking", and leave users an option to turn on these numeral exception checkings.
On 2018/07/20 22:19:46, Leonard Lausen <le...@lausen.nl> wrote:
> Hello MXNet community,
>
> It seems that there is currently no agreed upon principle to handle
> `nan` values in operators. This has led to inconsistencies between
> operators and also to inconsistency over releases. Some operators ignore
> nan values (eg. argmax), others treated it as maximum (e.g. topk up to
> mxnet v1.2) or just return “undefined” output (e.g. topk starting with
> mxnet v1.3).
>
> Initially the change in topk was reported as a bug
> (https://github.com/apache/incubator-mxnet/issues/8510) as some users
> relied on the behavior. However (and rightfully) @asmushetzel, who
> contributed the improved topk operator for mxnet v1.3 pointed out that
> the change did not break any documented behavior.
>
> To go forward, please share your opinion how MXNet should handle `nan`
> values. Should we continue to treat the behavior as undefined and
> possibly silently changing between releases? Should we define a
> reasonable standard (e.g. follow numpy) and treat operators that deviate
> as buggy? Should we just document how operators behave currently and
> warn if the behavior changes? Something else?
>
> Please make your opinion known so above issue can be resolved/closed and
> general guidelines can be defined for future contributions, following
> whatever consensus emerges.
>
> Thanks!
> Leonard
>
Re: How should MXNet treat nan values?
Posted by Leonard Lausen <l-...@lausen.nl>.
Thanks to everyone who made their opinion known. So far the consensus
is that any nan handling in MXNet should not affect performance, at
least not by default.
This still leaves the question open if we should aim for documenting the
behavior of MXNet operators under presence of nan values. For example,
should we include a sentence in the argmax and topk documentation?
Should the 1.3 release notes note the changed behavior of topk?
So far this has not been done. Instead any change of operator behavior
with respect to nan values is treated as implementation change that is
not worth noting to the user.
As this can decrease user experience, I advocate for documenting the
current behavior and possible future changes.
In case there are no objections, is there any way to edit the changelog
for the upcoming release?