You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uima.apache.org by Marshall Schor <ms...@schor.com> on 2009/03/10 23:49:41 UTC
Re: small memory footprint tradeoff configuration
After reviewing the previous chain of discussion on this topic, I would
like to start the next round, hopefully getting to some convergence :-).
1) On the topic of doing GC (garbage collection) versus copy to another
CAS - GC is conceptually perhaps less complex - you don't have mutliple
CASes around. I note that Adam (and maybe others) preferred this.
2) Keeping IDs: within the context of an aggregate, doing a GC and
having the resulting FeatureStructure IDs change is an issue. Some
suggestions to lessen this:
a) No automatic GC, just do when requested.
b) If done in an Aggregate (could be an inner one), guarantee that IDs
that existed upon entry to the aggregate would be preserved. This could
be done using the high-water-mark mechanism that was put in for
delta-cas. This seems to have some nice properties concerning what a
user of an aggregate has to know about the aggregate's inner behavior -
in particular, the user would not need to know about that aggregate's
call to GC, since the outer aggregate's "handles" to feature structures
would not move due to the GC.
Part of this concept could be to allow an option to have the GC to move
everything; maybe a different explicit call.
3) There is a potential to trade performance for space using String
"interning" - to insure string values set for features are stored just
once. This is typically done using a hashmap of some kind, so there's
overhead for that, so it may or may not actually reduce space. There is
also a potential to store Strings using UTF-8 encoding - which may or
may not save space (depends on the string, etc.)
For String interning, we could have two different kinds of approaches to
specifying its use: a global, application-wide setting or a specific
setting (e.g., add a new basic type to UIMA, called, for instance,
cas.uima.SharedString). Using a new type would allow users to pick just
the cases where they wanted the extra machinery to coalesce equal
strings to one shared object.
4) There is a potentially big space reduction possible by being able to
mark some fields of feature structures as never being "read". Such
fields could then be not stored. For instance, a feature structure of
type TOKEN might have many fields, representing various information -
only some of which might be used in a particular application. Even if
the ResultSpecification for a tokenizer is set to indicate not to
"produce" these fields, today, space is consumed for those fields (they
are filled with "null" or 0). If there are many instances of this
feature structure type (such as Token), this can be a significant space
saver. To identify a field as never being read, one could look at the
aggregate's component's capability specification - and mark the field if
it doesn't appear in any of the delegate's input specs. For an
outermost aggregate, one would probably want to add that aggregate's
output specification - to capture the outer application's potential use
of fields.
There is another layer of space granularity we could consider. This
would be to let the assembler divide the components into "groups"
(perhaps just using the natural grouping aggregates provide), and
compute (or specify) which fields of types were "not used" by group. For
instance, consider group1 which has lots of extra fields in Token, which
are used during group1 components, followed by group2 processing, which
doesn't use most of the fields in Token. A GC operation could then
"compress" the representation of Token (of those where preserving the
FeatureStructure ID wasn't required).
To avoid creation of new collections of components, we could restrict
the "groups" to be just what a particular aggregate contained. Using
this approach, we could envision using the input/output capability
specifications to automatically deduce which extra fields could be
eliminated. We could also have an automatic GC mode - which would
invoke the GC (that didn't alter pre-existing feature structures) at the
end of all aggregates. Although this might do too many gc's, it would
be conceptually simple.
-Marshall
Re: small memory footprint tradeoff configuration
Posted by Adam Lally <al...@alum.rpi.edu>.
On Fri, Mar 13, 2009 at 3:07 AM, Thilo Goetz <tw...@gmx.de> wrote:
> Marshall Schor wrote:
>> I agree with both of these concepts: only GC'ing things which are not
>> in the index and also not reachable from something that is in the index,
>> and making GC'ing (mostly) automatic, based on thresholds, etc, when a
>> component exits back to the framework. This would be fine for now - if
>> use cases come up where some more programmatic control of this is
>> needed, we could add something.
>>
>> Maybe the next thing to focus on is the "contract" re: GC running. For
>> a component (primitive or aggregate), the proposed contract is to have
>> the GC not change the FS "id"s that existed prior to the component
>> running. This is a tradeoff - for more stability with existing handle
>> uses, versus less "aggressive" GC's.
>
> The way I understood it, that was implied by Adam's proposal.
> If you only GC FSs that were created since the component was
> entered, there is no danger of changing the IDs. Wherever
> you actually do GC, you will have to change the FS IDs. GC
> without compaction is going to be useless.
>
> The FSs that were created by the app, for example, would never
> be garbage collected, so those references would stay valid until
> the end.
>
It's certainly possible, but more complicated, to have smarter
"handles" that get updated when the underlying FS gets moved in the
heap. Of course then the CAS would have to have a record of all the
outstanding handles, which it does not now have. But anyway, with my
proposal this complication would not be necessary.
-Adam
Re: small memory footprint tradeoff configuration
Posted by Marshall Schor <ms...@schor.com>.
Thilo Goetz wrote:
> Marshall Schor wrote:
>
>> I agree with both of these concepts: only GC'ing things which are not
>> in the index and also not reachable from something that is in the index,
>> and making GC'ing (mostly) automatic, based on thresholds, etc, when a
>> component exits back to the framework. This would be fine for now - if
>> use cases come up where some more programmatic control of this is
>> needed, we could add something.
>>
>> Maybe the next thing to focus on is the "contract" re: GC running. For
>> a component (primitive or aggregate), the proposed contract is to have
>> the GC not change the FS "id"s that existed prior to the component
>> running. This is a tradeoff - for more stability with existing handle
>> uses, versus less "aggressive" GC's.
>>
>
> The way I understood it, that was implied by Adam's proposal.
> If you only GC FSs that were created since the component was
> entered, there is no danger of changing the IDs. Wherever
> you actually do GC, you will have to change the FS IDs. GC
> without compaction is going to be useless.
>
+1 I missed that deduction ;-)
> The FSs that were created by the app, for example, would never
> be garbage collected, so those references would stay valid until
> the end.
>
true. I don't think this would be a significant issue.
This topic seems to have come to a pretty good concensus:
Control GC overall by some configuration parameter - default to
operate without it.
Have it occur after some component (primitive or aggregate) per some
(new?) spec to enable it.
Only have it affect things added by that component - preserving IDs
that surrounding components may be holding on to
Have it occur only if there is some likely gain from spending the time
to do it (if this can be figured out without much time overhead).
Please correct if this isn't your understanding of the approximate
concensus :-) . -Marshall
> --Thilo
>
>
>> -Marshall
>>
>
>
>
>
Re: small memory footprint tradeoff configuration
Posted by Thilo Goetz <tw...@gmx.de>.
Marshall Schor wrote:
> I agree with both of these concepts: only GC'ing things which are not
> in the index and also not reachable from something that is in the index,
> and making GC'ing (mostly) automatic, based on thresholds, etc, when a
> component exits back to the framework. This would be fine for now - if
> use cases come up where some more programmatic control of this is
> needed, we could add something.
>
> Maybe the next thing to focus on is the "contract" re: GC running. For
> a component (primitive or aggregate), the proposed contract is to have
> the GC not change the FS "id"s that existed prior to the component
> running. This is a tradeoff - for more stability with existing handle
> uses, versus less "aggressive" GC's.
The way I understood it, that was implied by Adam's proposal.
If you only GC FSs that were created since the component was
entered, there is no danger of changing the IDs. Wherever
you actually do GC, you will have to change the FS IDs. GC
without compaction is going to be useless.
The FSs that were created by the app, for example, would never
be garbage collected, so those references would stay valid until
the end.
--Thilo
>
> -Marshall
Re: small memory footprint tradeoff configuration
Posted by "D.J. McCloskey" <dj...@ie.ibm.com>.
I think this would be a good addition. I don't see a need for an explicit
call to invoke GC. The parameter of a threshold relative to CAS heapsize
would be useful. But also the ability to specify a GC on exiting one
analysis engine and before invoking the next. This could be turned off for
applications where folks don't want the overhead caused in complex
aggregates. Ideally, a combination of the threshold and the AE boundary
auto collect would also be possible by a specific values of the engine
boundary parameter. The idea here would be to only do the collect on the
boundary based on the heap/threshold ratio having been exceeded. This would
give more than enough control and addresses the need.
I don't know if it would have to be limited to a capability flow scenario
but it would be nice to also have the option to use the output capabilities
to define what to keep or was this what you were thinking?.
The notion of type driven collection is something I'd like to raise also,
often in analysis there are types which are purely supporting the
identification of some primary type. The ability to declare some types as
transient would perhaps aid the overall objective and mitigate in the
aggressive GC cases. On the aggressive GC point, I believe anything which
invalidates lowlevel handles is not really acceptable.
-DJ
-------------------
D.J McCloskey
IBM LanguageWare
IBM Ireland Product Distribution Limited registered in Ireland with number
92815. Registered office: Oldbrook House, 24-32 Pembroke Road,
Ballsbridge, Dublin 4
From: Marshall Schor <ms...@schor.com>
To: uima-dev@incubator.apache.org
Date: 12/03/2009 18:06
Subject: Re: small memory footprint tradeoff configuration
I agree with both of these concepts: only GC'ing things which are not
in the index and also not reachable from something that is in the index,
and making GC'ing (mostly) automatic, based on thresholds, etc, when a
component exits back to the framework. This would be fine for now - if
use cases come up where some more programmatic control of this is
needed, we could add something.
Maybe the next thing to focus on is the "contract" re: GC running. For
a component (primitive or aggregate), the proposed contract is to have
the GC not change the FS "id"s that existed prior to the component
running. This is a tradeoff - for more stability with existing handle
uses, versus less "aggressive" GC's.
-Marshall
Thilo Goetz wrote:
> Adam Lally wrote:
>
>> On Wed, Mar 11, 2009 at 8:53 AM, Marshall Schor <ms...@schor.com> wrote:
>>
>>> I agree in general about not making things more complicated at least to
>>> the user. I can imagine education working for
>>> 1) things like string interning
>>> 2) things like deleting features from type systems where they're not
>>> being used, and where the annotator producing them will respect this.
>>>
>>> What this approach seems to miss are the following kinds of things:
>>>
>>> 1) cases where some set of annotators produce feature structures,
which,
>>> after some point, are no longer needed, and are "deleted" but
>>> never-the-less continue to consume space.
>>>
>>> 2) cases where some set of annotators produce feature structures having
>>> lots of fields, where, after some point, the fields are no longer
needed.
>>>
>>> If these are not significant use-cases in practice, then I'm happy to
>>> think-about / work-on other things :-).
>>>
>>>
>> I'd like to propose discussing the different ideas here one at a time.
>> We had enough trouble coming to any agreement on GC the last time
>> that we discussed it, without also throwing string interning and
>> feature deleting into the mix.
>>
>> So focusing on GC first (unless you think one of the others is more
important):
>>
>> My inclination is to assure that GC deletes only garbage, and that
>> there's no possibility that anything GC'ed could have been referenced
>> by anybody. The other proposals that don't have this guarantee are
>> scary to me.
>>
>> A way to accomplish this guarantee would be that when the process
>> method of an AnalysisEngine (could be either primitive or aggregate)
>> completes, we can mark as garbage any FS's that were created since the
>> beginning of that process method, but which are not referenced
>> directly or indirectly from anything in the indexes. Does this
>> concept seem reasonable?
>>
>
> +1. I like the idea because it is sort of local on the one
> hand, but still allows one to delete FSs from indexes
> later in the processing and have them garbage collected
> (on exiting the containing aggregate).
>
>
>> The next question is under what conditions would a GC execute.
>> Requiring an explicit call seems counter to what other garbage
>> collecting runtime environments do, and like Thilo I'm confused about
>> who would call this and when. I think it would be better to define
>> the parameters that control GC in the PerformanceTuningSettings that
>> we already have, and make them dependent on how much CAS heap space is
>> used relative to a GC threshold that the user has set in the
>> PerformanceTuningSettings.
>>
>
> +1, and the default could be "no GC", so it would be
> perfectly backwards compatible. I'm thinking of the
> kinds of scenarios that I often work with, where
> basically all the annotations are later written to
> an index, and any attempt at GC would be futile and
> just consume time to no benefit.
>
>
>> -Adam
>>
>
>
>
>
Re: small memory footprint tradeoff configuration
Posted by Marshall Schor <ms...@schor.com>.
I agree with both of these concepts: only GC'ing things which are not
in the index and also not reachable from something that is in the index,
and making GC'ing (mostly) automatic, based on thresholds, etc, when a
component exits back to the framework. This would be fine for now - if
use cases come up where some more programmatic control of this is
needed, we could add something.
Maybe the next thing to focus on is the "contract" re: GC running. For
a component (primitive or aggregate), the proposed contract is to have
the GC not change the FS "id"s that existed prior to the component
running. This is a tradeoff - for more stability with existing handle
uses, versus less "aggressive" GC's.
-Marshall
Thilo Goetz wrote:
> Adam Lally wrote:
>
>> On Wed, Mar 11, 2009 at 8:53 AM, Marshall Schor <ms...@schor.com> wrote:
>>
>>> I agree in general about not making things more complicated at least to
>>> the user. I can imagine education working for
>>> 1) things like string interning
>>> 2) things like deleting features from type systems where they're not
>>> being used, and where the annotator producing them will respect this.
>>>
>>> What this approach seems to miss are the following kinds of things:
>>>
>>> 1) cases where some set of annotators produce feature structures, which,
>>> after some point, are no longer needed, and are "deleted" but
>>> never-the-less continue to consume space.
>>>
>>> 2) cases where some set of annotators produce feature structures having
>>> lots of fields, where, after some point, the fields are no longer needed.
>>>
>>> If these are not significant use-cases in practice, then I'm happy to
>>> think-about / work-on other things :-).
>>>
>>>
>> I'd like to propose discussing the different ideas here one at a time.
>> We had enough trouble coming to any agreement on GC the last time
>> that we discussed it, without also throwing string interning and
>> feature deleting into the mix.
>>
>> So focusing on GC first (unless you think one of the others is more important):
>>
>> My inclination is to assure that GC deletes only garbage, and that
>> there's no possibility that anything GC'ed could have been referenced
>> by anybody. The other proposals that don't have this guarantee are
>> scary to me.
>>
>> A way to accomplish this guarantee would be that when the process
>> method of an AnalysisEngine (could be either primitive or aggregate)
>> completes, we can mark as garbage any FS's that were created since the
>> beginning of that process method, but which are not referenced
>> directly or indirectly from anything in the indexes. Does this
>> concept seem reasonable?
>>
>
> +1. I like the idea because it is sort of local on the one
> hand, but still allows one to delete FSs from indexes
> later in the processing and have them garbage collected
> (on exiting the containing aggregate).
>
>
>> The next question is under what conditions would a GC execute.
>> Requiring an explicit call seems counter to what other garbage
>> collecting runtime environments do, and like Thilo I'm confused about
>> who would call this and when. I think it would be better to define
>> the parameters that control GC in the PerformanceTuningSettings that
>> we already have, and make them dependent on how much CAS heap space is
>> used relative to a GC threshold that the user has set in the
>> PerformanceTuningSettings.
>>
>
> +1, and the default could be "no GC", so it would be
> perfectly backwards compatible. I'm thinking of the
> kinds of scenarios that I often work with, where
> basically all the annotations are later written to
> an index, and any attempt at GC would be futile and
> just consume time to no benefit.
>
>
>> -Adam
>>
>
>
>
>
Re: small memory footprint tradeoff configuration
Posted by Thilo Goetz <tw...@gmx.de>.
Adam Lally wrote:
> On Wed, Mar 11, 2009 at 8:53 AM, Marshall Schor <ms...@schor.com> wrote:
>> I agree in general about not making things more complicated at least to
>> the user. I can imagine education working for
>> 1) things like string interning
>> 2) things like deleting features from type systems where they're not
>> being used, and where the annotator producing them will respect this.
>>
>> What this approach seems to miss are the following kinds of things:
>>
>> 1) cases where some set of annotators produce feature structures, which,
>> after some point, are no longer needed, and are "deleted" but
>> never-the-less continue to consume space.
>>
>> 2) cases where some set of annotators produce feature structures having
>> lots of fields, where, after some point, the fields are no longer needed.
>>
>> If these are not significant use-cases in practice, then I'm happy to
>> think-about / work-on other things :-).
>>
>
>
> I'd like to propose discussing the different ideas here one at a time.
> We had enough trouble coming to any agreement on GC the last time
> that we discussed it, without also throwing string interning and
> feature deleting into the mix.
>
> So focusing on GC first (unless you think one of the others is more important):
>
> My inclination is to assure that GC deletes only garbage, and that
> there's no possibility that anything GC'ed could have been referenced
> by anybody. The other proposals that don't have this guarantee are
> scary to me.
>
> A way to accomplish this guarantee would be that when the process
> method of an AnalysisEngine (could be either primitive or aggregate)
> completes, we can mark as garbage any FS's that were created since the
> beginning of that process method, but which are not referenced
> directly or indirectly from anything in the indexes. Does this
> concept seem reasonable?
+1. I like the idea because it is sort of local on the one
hand, but still allows one to delete FSs from indexes
later in the processing and have them garbage collected
(on exiting the containing aggregate).
>
> The next question is under what conditions would a GC execute.
> Requiring an explicit call seems counter to what other garbage
> collecting runtime environments do, and like Thilo I'm confused about
> who would call this and when. I think it would be better to define
> the parameters that control GC in the PerformanceTuningSettings that
> we already have, and make them dependent on how much CAS heap space is
> used relative to a GC threshold that the user has set in the
> PerformanceTuningSettings.
+1, and the default could be "no GC", so it would be
perfectly backwards compatible. I'm thinking of the
kinds of scenarios that I often work with, where
basically all the annotations are later written to
an index, and any attempt at GC would be futile and
just consume time to no benefit.
>
> -Adam
Re: small memory footprint tradeoff configuration
Posted by Eddie Epstein <ea...@gmail.com>.
On Fri, Mar 13, 2009 at 10:03 AM, Adam Lally <al...@alum.rpi.edu> wrote:
>>
>> One user scenario that motivated this thread was that an aggregate designer
>> knew exactly when they wanted to do GC. What is wrong with giving them
>> a CAS method to call?
>>
>
> Where would they call this method from? An aggregate designer would
> just write a descriptor, not code. I suppose they could stick in a
> "GC annotator" that does nothing but call the GC method, but this
> seems like a hack.
OK. Another problem with a GC call is that FS handles within that
component could
be broken, so better not to allow that possibility.
> Can this be handled with configuration? I think somebody suggested
> something like this already, but I can't find the quote right now.
The quote is from DJ, I think.
> One of the GC configuration options would be to specify that a GC
> should always occur after a particular delegate AE has finished (or
> that GC should occur after that AE finished, if the heap size is also
> above a certain threshold).
Tong put a proposal in
http://cwiki.apache.org/UIMA/improving-uima-debug-capabilities.html
which includes an "operational descriptor" to control runtime behaviors.
Maybe that would be useful for controlling GC.
Eddie
Re: small memory footprint tradeoff configuration
Posted by Adam Lally <al...@alum.rpi.edu>.
On Thu, Mar 12, 2009 at 3:49 PM, Eddie Epstein <ea...@gmail.com> wrote:
> On Thu, Mar 12, 2009 at 12:14 PM, Adam Lally <al...@alum.rpi.edu> wrote:
>> The next question is under what conditions would a GC execute.
>> Requiring an explicit call seems counter to what other garbage
>> collecting runtime environments do, and like Thilo I'm confused about
>> who would call this and when. I think it would be better to define
>> the parameters that control GC in the PerformanceTuningSettings that
>> we already have, and make them dependent on how much CAS heap space is
>> used relative to a GC threshold that the user has set in the
>> PerformanceTuningSettings.
>
> Given the current CAS implementation, GC related operations are going to
> be fairly expensive, for example, even just computing how much data is
> available to be deleted. So I'd be concerned that any automatic GC based
> on CAS size or other dynamic properties may often be CPU costly.
>
> One user scenario that motivated this thread was that an aggregate designer
> knew exactly when they wanted to do GC. What is wrong with giving them
> a CAS method to call?
>
Where would they call this method from? An aggregate designer would
just write a descriptor, not code. I suppose they could stick in a
"GC annotator" that does nothing but call the GC method, but this
seems like a hack.
Can this be handled with configuration? I think somebody suggested
something like this already, but I can't find the quote right now.
One of the GC configuration options would be to specify that a GC
should always occur after a particular delegate AE has finished (or
that GC should occur after that AE finished, if the heap size is also
above a certain threshold).
-Adam
Re: small memory footprint tradeoff configuration
Posted by Eddie Epstein <ea...@gmail.com>.
On Thu, Mar 12, 2009 at 12:14 PM, Adam Lally <al...@alum.rpi.edu> wrote:
> The next question is under what conditions would a GC execute.
> Requiring an explicit call seems counter to what other garbage
> collecting runtime environments do, and like Thilo I'm confused about
> who would call this and when. I think it would be better to define
> the parameters that control GC in the PerformanceTuningSettings that
> we already have, and make them dependent on how much CAS heap space is
> used relative to a GC threshold that the user has set in the
> PerformanceTuningSettings.
Given the current CAS implementation, GC related operations are going to
be fairly expensive, for example, even just computing how much data is
available to be deleted. So I'd be concerned that any automatic GC based
on CAS size or other dynamic properties may often be CPU costly.
One user scenario that motivated this thread was that an aggregate designer
knew exactly when they wanted to do GC. What is wrong with giving them
a CAS method to call?
Re: small memory footprint tradeoff configuration
Posted by Adam Lally <al...@alum.rpi.edu>.
On Wed, Mar 11, 2009 at 8:53 AM, Marshall Schor <ms...@schor.com> wrote:
> I agree in general about not making things more complicated at least to
> the user. I can imagine education working for
> 1) things like string interning
> 2) things like deleting features from type systems where they're not
> being used, and where the annotator producing them will respect this.
>
> What this approach seems to miss are the following kinds of things:
>
> 1) cases where some set of annotators produce feature structures, which,
> after some point, are no longer needed, and are "deleted" but
> never-the-less continue to consume space.
>
> 2) cases where some set of annotators produce feature structures having
> lots of fields, where, after some point, the fields are no longer needed.
>
> If these are not significant use-cases in practice, then I'm happy to
> think-about / work-on other things :-).
>
I'd like to propose discussing the different ideas here one at a time.
We had enough trouble coming to any agreement on GC the last time
that we discussed it, without also throwing string interning and
feature deleting into the mix.
So focusing on GC first (unless you think one of the others is more important):
My inclination is to assure that GC deletes only garbage, and that
there's no possibility that anything GC'ed could have been referenced
by anybody. The other proposals that don't have this guarantee are
scary to me.
A way to accomplish this guarantee would be that when the process
method of an AnalysisEngine (could be either primitive or aggregate)
completes, we can mark as garbage any FS's that were created since the
beginning of that process method, but which are not referenced
directly or indirectly from anything in the indexes. Does this
concept seem reasonable?
The next question is under what conditions would a GC execute.
Requiring an explicit call seems counter to what other garbage
collecting runtime environments do, and like Thilo I'm confused about
who would call this and when. I think it would be better to define
the parameters that control GC in the PerformanceTuningSettings that
we already have, and make them dependent on how much CAS heap space is
used relative to a GC threshold that the user has set in the
PerformanceTuningSettings.
-Adam
Re: small memory footprint tradeoff configuration
Posted by Marshall Schor <ms...@schor.com>.
Thilo Goetz wrote:
> Marshall Schor wrote:
> [...]
>
>> I agree that backward compatibility is important and is an issue. To
>> help the transition to this new scheme, I think an overall global switch
>> is needed (similar to the switches we have for JCas "interning") that
>> would by default make things work the way they do now. A user
>> interested in small-footprint operation (and in trading off some
>> additional processing cycles to achieve it) would enable this switch.
>>
>> To help it "work" - we would allow things to continue to operation which
>> "set" a non-stored feature - theset would just become no-ops. Then if
>> the annotator wasn't paying attention to ResultSpecification, and tried
>> to set features that were not used, it would still work.
>>
>> On the other end, if an annotator actually made use of a particular
>> feature, but didn't specify it in its "input capability specification",
>> that would fail with this scheme. The failure would be some kind of
>> Java exception, which would probably be noticed. To recover, a user of
>> such a component would modify the input capability specification to
>> indicate that that feature was needed.
>>
>
> If a feature is defined in the type system, it should be there
> for the annotator writer to use. Who are we to know how people
> will use those features?
>
>
>> As I write this, I notice that the input capability specification for a
>> primitive annotator doesn't quite fit the meaning hear - because I think
>> it means that this annotator needs that feature upon input - and this
>> edge case - where the annotator itself produces this feature, and then
>> also uses it - is not part of that definition. We could either expand
>> the meaning here to include this edge case, or (possibly a better
>> option) introduce, explicitly, another piece of metadata indicating that
>> a particular type/field was both created and used by this one primitive
>> annotator. A third option could be to store these "unused" features if
>> set (in some out-of-line temporary storage) for the duration of the
>> running of a particular annotator, just in case these were "used" by the
>> same annotator, and then discard that extra storage after the annotator
>> exits. This would be a big (but temporary) storage hit, though, so I
>> don't think I would want to do this.
>>
>
> I vote we don't make things even more complicated than they
> already are, and educate those people who need a performance
> boost.
>
I agree in general about not making things more complicated at least to
the user. I can imagine education working for
1) things like string interning
2) things like deleting features from type systems where they're not
being used, and where the annotator producing them will respect this.
What this approach seems to miss are the following kinds of things:
1) cases where some set of annotators produce feature structures, which,
after some point, are no longer needed, and are "deleted" but
never-the-less continue to consume space.
2) cases where some set of annotators produce feature structures having
lots of fields, where, after some point, the fields are no longer needed.
If these are not significant use-cases in practice, then I'm happy to
think-about / work-on other things :-).
-Marshall
> --Thilo
>
>
>
>
Re: small memory footprint tradeoff configuration
Posted by Thilo Goetz <tw...@gmx.de>.
Marshall Schor wrote:
[...]
> I agree that backward compatibility is important and is an issue. To
> help the transition to this new scheme, I think an overall global switch
> is needed (similar to the switches we have for JCas "interning") that
> would by default make things work the way they do now. A user
> interested in small-footprint operation (and in trading off some
> additional processing cycles to achieve it) would enable this switch.
>
> To help it "work" - we would allow things to continue to operation which
> "set" a non-stored feature - theset would just become no-ops. Then if
> the annotator wasn't paying attention to ResultSpecification, and tried
> to set features that were not used, it would still work.
>
> On the other end, if an annotator actually made use of a particular
> feature, but didn't specify it in its "input capability specification",
> that would fail with this scheme. The failure would be some kind of
> Java exception, which would probably be noticed. To recover, a user of
> such a component would modify the input capability specification to
> indicate that that feature was needed.
If a feature is defined in the type system, it should be there
for the annotator writer to use. Who are we to know how people
will use those features?
>
> As I write this, I notice that the input capability specification for a
> primitive annotator doesn't quite fit the meaning hear - because I think
> it means that this annotator needs that feature upon input - and this
> edge case - where the annotator itself produces this feature, and then
> also uses it - is not part of that definition. We could either expand
> the meaning here to include this edge case, or (possibly a better
> option) introduce, explicitly, another piece of metadata indicating that
> a particular type/field was both created and used by this one primitive
> annotator. A third option could be to store these "unused" features if
> set (in some out-of-line temporary storage) for the duration of the
> running of a particular annotator, just in case these were "used" by the
> same annotator, and then discard that extra storage after the annotator
> exits. This would be a big (but temporary) storage hit, though, so I
> don't think I would want to do this.
I vote we don't make things even more complicated than they
already are, and educate those people who need a performance
boost.
--Thilo
Re: small memory footprint tradeoff configuration
Posted by Marshall Schor <ms...@schor.com>.
Thanks for your comments.
Thilo Goetz wrote:
> Marshall Schor wrote:
>
>> After reviewing the previous chain of discussion on this topic, I would
>> like to start the next round, hopefully getting to some convergence :-).
>>
>> 1) On the topic of doing GC (garbage collection) versus copy to another
>> CAS - GC is conceptually perhaps less complex - you don't have mutliple
>> CASes around. I note that Adam (and maybe others) preferred this.
>>
>> 2) Keeping IDs: within the context of an aggregate, doing a GC and
>> having the resulting FeatureStructure IDs change is an issue. Some
>> suggestions to lessen this:
>> a) No automatic GC, just do when requested.
>> b) If done in an Aggregate (could be an inner one), guarantee that IDs
>> that existed upon entry to the aggregate would be preserved. This could
>> be done using the high-water-mark mechanism that was put in for
>> delta-cas. This seems to have some nice properties concerning what a
>> user of an aggregate has to know about the aggregate's inner behavior -
>> in particular, the user would not need to know about that aggregate's
>> call to GC, since the outer aggregate's "handles" to feature structures
>> would not move due to the GC.
>>
>
> I'm a little foggy on the concept of explicit GC calls. Where would
> the API reside, and who would be allowed to call it, and when?
>
One possibility is to not have it be callable, but to have an annotator
or aggregate marked so that it would be called by the framework after a
CAS exited that component. I haven't figured out other details here,
though. Details might include (a) an overall enabling of this, (b) some
indication that there's enough to reclaim in the CAS to make it
worthwhile. Worthwhile may be hard to define, though, so some kind of
explicit indication to do it might be needed. The indication might be
to set some context flag saying to do the GC when you exit the
annotator. Here's a use case: testing shows your application is taking
31 MB of storage, but the app in which you're embedded has a hard limit
of 30 MB...
>
>> Part of this concept could be to allow an option to have the GC to move
>> everything; maybe a different explicit call.
>>
>> 3) There is a potential to trade performance for space using String
>> "interning" - to insure string values set for features are stored just
>> once. This is typically done using a hashmap of some kind, so there's
>> overhead for that, so it may or may not actually reduce space. There is
>> also a potential to store Strings using UTF-8 encoding - which may or
>> may not save space (depends on the string, etc.)
>>
>
> Any Java programmer who does a lot of string handling should know
> when to intern strings, when to use constants, and when to create
> new Strings.
Well, my experience is quite different. Many Java programmers around
here were unfamiliar with interning. But I do basically agree that some
(or most) of this benefit can happen via annotator writers. Perhaps we
need to document this in some new section (e.g. on how to write small
footprint annotators).
> Who do you think you're going to be helping with this?
> What's the use case?
>> For String interning, we could have two different kinds of approaches to
>> specifying its use: a global, application-wide setting or a specific
>> setting (e.g., add a new basic type to UIMA, called, for instance,
>> cas.uima.SharedString). Using a new type would allow users to pick just
>> the cases where they wanted the extra machinery to coalesce equal
>> strings to one shared object.
>>
>> 4) There is a potentially big space reduction possible by being able to
>> mark some fields of feature structures as never being "read". Such
>> fields could then be not stored. For instance, a feature structure of
>> type TOKEN might have many fields, representing various information -
>> only some of which might be used in a particular application. Even if
>> the ResultSpecification for a tokenizer is set to indicate not to
>> "produce" these fields, today, space is consumed for those fields (they
>> are filled with "null" or 0). If there are many instances of this
>> feature structure type (such as Token), this can be a significant space
>> saver. To identify a field as never being read, one could look at the
>> aggregate's component's capability specification - and mark the field if
>> it doesn't appear in any of the delegate's input specs. For an
>> outermost aggregate, one would probably want to add that aggregate's
>> output specification - to capture the outer application's potential use
>> of fields.
>>
>
> We need to be careful here not to destroy backward compatibility.
> Result specs are optional in fixed flows, and many annotators (that
> I use) don't use them.
>
> What we usually do in cases like this is that we modify the type
> system. The annotator (in this case, the tokenizer) checks the
> type system on startup. Presence/absence of features triggers/
> inhibits certain processing. This may not be an ideal solution,
> but it works because it requires the cooperation of the annotator
> writer.
>
> If you don't have access to the annotator's source code, you'll
> never know if it can really work without those features in all
> cases. If you do have the source code, you can make it work
> with the scheme above.
>
I agree that backward compatibility is important and is an issue. To
help the transition to this new scheme, I think an overall global switch
is needed (similar to the switches we have for JCas "interning") that
would by default make things work the way they do now. A user
interested in small-footprint operation (and in trading off some
additional processing cycles to achieve it) would enable this switch.
To help it "work" - we would allow things to continue to operation which
"set" a non-stored feature - theset would just become no-ops. Then if
the annotator wasn't paying attention to ResultSpecification, and tried
to set features that were not used, it would still work.
On the other end, if an annotator actually made use of a particular
feature, but didn't specify it in its "input capability specification",
that would fail with this scheme. The failure would be some kind of
Java exception, which would probably be noticed. To recover, a user of
such a component would modify the input capability specification to
indicate that that feature was needed.
As I write this, I notice that the input capability specification for a
primitive annotator doesn't quite fit the meaning hear - because I think
it means that this annotator needs that feature upon input - and this
edge case - where the annotator itself produces this feature, and then
also uses it - is not part of that definition. We could either expand
the meaning here to include this edge case, or (possibly a better
option) introduce, explicitly, another piece of metadata indicating that
a particular type/field was both created and used by this one primitive
annotator. A third option could be to store these "unused" features if
set (in some out-of-line temporary storage) for the duration of the
running of a particular annotator, just in case these were "used" by the
same annotator, and then discard that extra storage after the annotator
exits. This would be a big (but temporary) storage hit, though, so I
don't think I would want to do this.
-Marshall
>
>> There is another layer of space granularity we could consider. This
>> would be to let the assembler divide the components into "groups"
>> (perhaps just using the natural grouping aggregates provide), and
>> compute (or specify) which fields of types were "not used" by group. For
>> instance, consider group1 which has lots of extra fields in Token, which
>> are used during group1 components, followed by group2 processing, which
>> doesn't use most of the fields in Token. A GC operation could then
>> "compress" the representation of Token (of those where preserving the
>> FeatureStructure ID wasn't required).
>>
>> To avoid creation of new collections of components, we could restrict
>> the "groups" to be just what a particular aggregate contained. Using
>> this approach, we could envision using the input/output capability
>> specifications to automatically deduce which extra fields could be
>> eliminated. We could also have an automatic GC mode - which would
>> invoke the GC (that didn't alter pre-existing feature structures) at the
>> end of all aggregates. Although this might do too many gc's, it would
>> be conceptually simple.
>>
>> -Marshall
>>
>
>
>
Re: small memory footprint tradeoff configuration
Posted by Thilo Goetz <tw...@gmx.de>.
Marshall Schor wrote:
> After reviewing the previous chain of discussion on this topic, I would
> like to start the next round, hopefully getting to some convergence :-).
>
> 1) On the topic of doing GC (garbage collection) versus copy to another
> CAS - GC is conceptually perhaps less complex - you don't have mutliple
> CASes around. I note that Adam (and maybe others) preferred this.
>
> 2) Keeping IDs: within the context of an aggregate, doing a GC and
> having the resulting FeatureStructure IDs change is an issue. Some
> suggestions to lessen this:
> a) No automatic GC, just do when requested.
> b) If done in an Aggregate (could be an inner one), guarantee that IDs
> that existed upon entry to the aggregate would be preserved. This could
> be done using the high-water-mark mechanism that was put in for
> delta-cas. This seems to have some nice properties concerning what a
> user of an aggregate has to know about the aggregate's inner behavior -
> in particular, the user would not need to know about that aggregate's
> call to GC, since the outer aggregate's "handles" to feature structures
> would not move due to the GC.
I'm a little foggy on the concept of explicit GC calls. Where would
the API reside, and who would be allowed to call it, and when?
>
> Part of this concept could be to allow an option to have the GC to move
> everything; maybe a different explicit call.
>
> 3) There is a potential to trade performance for space using String
> "interning" - to insure string values set for features are stored just
> once. This is typically done using a hashmap of some kind, so there's
> overhead for that, so it may or may not actually reduce space. There is
> also a potential to store Strings using UTF-8 encoding - which may or
> may not save space (depends on the string, etc.)
Any Java programmer who does a lot of string handling should know
when to intern strings, when to use constants, and when to create
new Strings. Who do you think you're going to be helping with this?
What's the use case?
>
> For String interning, we could have two different kinds of approaches to
> specifying its use: a global, application-wide setting or a specific
> setting (e.g., add a new basic type to UIMA, called, for instance,
> cas.uima.SharedString). Using a new type would allow users to pick just
> the cases where they wanted the extra machinery to coalesce equal
> strings to one shared object.
>
> 4) There is a potentially big space reduction possible by being able to
> mark some fields of feature structures as never being "read". Such
> fields could then be not stored. For instance, a feature structure of
> type TOKEN might have many fields, representing various information -
> only some of which might be used in a particular application. Even if
> the ResultSpecification for a tokenizer is set to indicate not to
> "produce" these fields, today, space is consumed for those fields (they
> are filled with "null" or 0). If there are many instances of this
> feature structure type (such as Token), this can be a significant space
> saver. To identify a field as never being read, one could look at the
> aggregate's component's capability specification - and mark the field if
> it doesn't appear in any of the delegate's input specs. For an
> outermost aggregate, one would probably want to add that aggregate's
> output specification - to capture the outer application's potential use
> of fields.
We need to be careful here not to destroy backward compatibility.
Result specs are optional in fixed flows, and many annotators (that
I use) don't use them.
What we usually do in cases like this is that we modify the type
system. The annotator (in this case, the tokenizer) checks the
type system on startup. Presence/absence of features triggers/
inhibits certain processing. This may not be an ideal solution,
but it works because it requires the cooperation of the annotator
writer.
If you don't have access to the annotator's source code, you'll
never know if it can really work without those features in all
cases. If you do have the source code, you can make it work
with the scheme above.
>
> There is another layer of space granularity we could consider. This
> would be to let the assembler divide the components into "groups"
> (perhaps just using the natural grouping aggregates provide), and
> compute (or specify) which fields of types were "not used" by group. For
> instance, consider group1 which has lots of extra fields in Token, which
> are used during group1 components, followed by group2 processing, which
> doesn't use most of the fields in Token. A GC operation could then
> "compress" the representation of Token (of those where preserving the
> FeatureStructure ID wasn't required).
>
> To avoid creation of new collections of components, we could restrict
> the "groups" to be just what a particular aggregate contained. Using
> this approach, we could envision using the input/output capability
> specifications to automatically deduce which extra fields could be
> eliminated. We could also have an automatic GC mode - which would
> invoke the GC (that didn't alter pre-existing feature structures) at the
> end of all aggregates. Although this might do too many gc's, it would
> be conceptually simple.
>
> -Marshall