[DISCUSS] Result Specs, cap. lang. flow, was: Re: Default Result Specifications too complicated?

[Thilo Goetz <tw...@gmx.de>]

As email discussion on this subject was dragging on, and the
time for our planned next release is fast approaching, the UIMA
committers had a phone call to resolve this issue.  Present were:

Adam, Eddie, Marshall, Thilo (Michael is currently unavailable,
back next week)

We had a lively discussion around the topics that are already
mentioned in the email thread.  We finally reached consensus that

a) The current design and implementation of result specifications
is unsatisfactory, for a variety of reasons.  One reason is that
many users are completely unaware of the concept of result
specifications, which can lead to the kind of issues raised by
Adam in the email that started this discussion, a main one being
the difficulty of debugging when annotators fail to produce
expected results. Another important issue is that currently, result
specifications are not passed on to remotely deployed processors,
making the behavior of analysis different for integrated vs. remote
deployment of analysis components.  At the same time, the issue that
result specifications are trying to address, namely the dynamic
configuration of analysis components, is an important one.  So it
would be good if there was a standardized way to address this
concern.  One idea that arose was to put the result spec information
in the CAS directly (which would also fix the remote deployment
issue).  Since this idea is of general concern and wider reach, we
are proposing to raise the issue with the OASIS TC and there try to
find a general solution that will then be implemented in some future
version of Apache UIMA.

b) To address the concern of backward compatibility - fixing the
incompatibility of the capability-language-flow (CLF) between version
1.x and 2.x of UIMA, we propose to add to the flow controller the
deprecated ability to set the result specification of analysis
engines.  We will use this API to make the current implementation of
the CLF behave the same way wrt result specs as the 1.x version.  We
are aware that this may possibly lead to issues for current 2.x users
who use the CLF.  However, we think that there are probably very few
or no such users, and even for those, the change will very likely have
no consequences.  Still, we need to point this out in the changelog.
We will not advertise this ability (of the flow controller to set the
result spec) in our user documentation, and in the code make it clear
that it is there for backward compatibility in the CLF only,  pending
a longer-term re-design in this area.

In the long term, we expect the result specs to be redesigned and
perhaps replaced by a more general mechanism.  At that point, any
application or analytic that uses result specs may have to change.  We
expect this will occur when the OASIS UIMA standard is finalized, as
the version of Apache UIMA that implements the standard will likely
have more changes that would require updates to user code.

This email documents the discussion we had; we now invite further
discussion and opinions on this, so we can come to a rapid conclusion
for our upcoming release.

--Thilo

Re: [DISCUSS] Result Specs, cap. lang. flow, was: Re: Default Result Specifications too complicated?

[Marshall Schor <ms...@schor.com>]

+1 

Backwards compatibility is something we take seriously, and when a 
change is needed,
we should provide transition time for our existing users. 
Results specifications seem to be good concept - and this approach
gives us time to thoughtfully evolve it.

-Marshall

Thilo Goetz wrote:
> As email discussion on this subject was dragging on, and the
> time for our planned next release is fast approaching, the UIMA
> committers had a phone call to resolve this issue.  Present were:
>
> Adam, Eddie, Marshall, Thilo (Michael is currently unavailable,
> back next week)
>
> We had a lively discussion around the topics that are already
> mentioned in the email thread.  We finally reached consensus that
>
> a) The current design and implementation of result specifications
> is unsatisfactory, for a variety of reasons.  One reason is that
> many users are completely unaware of the concept of result
> specifications, which can lead to the kind of issues raised by
> Adam in the email that started this discussion, a main one being
> the difficulty of debugging when annotators fail to produce
> expected results. Another important issue is that currently, result
> specifications are not passed on to remotely deployed processors,
> making the behavior of analysis different for integrated vs. remote
> deployment of analysis components.  At the same time, the issue that
> result specifications are trying to address, namely the dynamic
> configuration of analysis components, is an important one.  So it
> would be good if there was a standardized way to address this
> concern.  One idea that arose was to put the result spec information
> in the CAS directly (which would also fix the remote deployment
> issue).  Since this idea is of general concern and wider reach, we
> are proposing to raise the issue with the OASIS TC and there try to
> find a general solution that will then be implemented in some future
> version of Apache UIMA.
>
> b) To address the concern of backward compatibility - fixing the
> incompatibility of the capability-language-flow (CLF) between version
> 1.x and 2.x of UIMA, we propose to add to the flow controller the
> deprecated ability to set the result specification of analysis
> engines.  We will use this API to make the current implementation of
> the CLF behave the same way wrt result specs as the 1.x version.  We
> are aware that this may possibly lead to issues for current 2.x users
> who use the CLF.  However, we think that there are probably very few
> or no such users, and even for those, the change will very likely have
> no consequences.  Still, we need to point this out in the changelog.
> We will not advertise this ability (of the flow controller to set the
> result spec) in our user documentation, and in the code make it clear
> that it is there for backward compatibility in the CLF only,  pending
> a longer-term re-design in this area.
>
> In the long term, we expect the result specs to be redesigned and
> perhaps replaced by a more general mechanism.  At that point, any
> application or analytic that uses result specs may have to change.  We
> expect this will occur when the OASIS UIMA standard is finalized, as
> the version of Apache UIMA that implements the standard will likely
> have more changes that would require updates to user code.
>
> This email documents the discussion we had; we now invite further
> discussion and opinions on this, so we can come to a rapid conclusion
> for our upcoming release.
>
> --Thilo
>
>
>
>
>