You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Ken Geis <kg...@speakeasy.org> on 2004/05/11 22:56:23 UTC
[Math] Cloning statistics
I'm playing with commons-math to implement a data mining algorithm and I
am having a performance problem.
I am doing running statistics over an ordered set of data, storing the
statistics at each new value I come across. One way of doing this would
be to have an array of SummaryStatistics and do
for (int i = 0; i < length; i++)
{
for (int j = i; j < length; j++)
{
statsArray[j].addValue(values[i]);
}
}
another way is to do
for (int i = 0; i < length; i++)
{
stats.addValue(values[i]);
statsArray[i] = SerializationUtils.clone(stats);
}
A lot of these objects are marked Serializable, but clone methods do not
exist. That's why I use commons-lang SerializationUtils.
Unfortunately, that makes the cloning take up 50% of my runtime because
(de)serialization is expensive.
I will probably patch the statistics classes, implementing enough of
clone() to make me happy. Would you like this patch?
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [Math] Cloning statistics
Posted by Phil Steitz <ph...@steitz.com>.
Mark R. Diggory wrote:
>
>
> Phil Steitz wrote:
>
>> Ken Geis wrote:
>>
>>> I'm playing with commons-math to implement a data mining algorithm
>>> and I am having a performance problem.
>>>
>>> I am doing running statistics over an ordered set of data, storing
>>> the statistics at each new value I come across. One way of doing
>>> this would be to have an array of SummaryStatistics and do
>>>
>>> for (int i = 0; i < length; i++)
>>> {
>>> for (int j = i; j < length; j++)
>>> {
>>> statsArray[j].addValue(values[i]);
>>> }
>>> }
>>>
>>> another way is to do
>>>
>>> for (int i = 0; i < length; i++)
>>> {
>>> stats.addValue(values[i]);
>>> statsArray[i] = SerializationUtils.clone(stats);
>>> }
>>>
>>> A lot of these objects are marked Serializable, but clone methods do
>>> not exist. That's why I use commons-lang SerializationUtils.
>>> Unfortunately, that makes the cloning take up 50% of my runtime
>>> because (de)serialization is expensive.
>>>
>>> I will probably patch the statistics classes, implementing enough of
>>> clone() to make me happy. Would you like this patch?
>>
>>
>>
>> An efficient cloning method might be useful, but it would still carry
>> around extra baggage and overhead for your use case
>> (SummaryStatisticsImpl nests a bunch of little stats objects and
>> other instance data).
>>
>> What we might want to do is add a StatisticalSummaryBean, implementing
>> the StatisticalSummary interface and add a getSummary method to
>> SummaryStatistics returning an instance of this "value bean"
>> containing only the values of the statistics. Then you could just do
>>
>> for (int i = 0; i < length; i++)
>> {
>> stats.addValue(values[i]);
>> statsArray[i] = stats.getSummary();
>> }
>>
>> since I presume that all you will want to do with statsArray[i] is
>> things like getMean(), getVariance(), etc. This would require much
>> less overhead than cloning the whole SummaryStatisticsImpl instance
>> each time.
>>
>> Since this would amount to a change to the SummaryStatistics
>> interface, if we want to do it, we should do it now, before 1.0. I am
>> +1 to this change and willing to implement it if no one objects.
>>
>> Phil
>
>
> Phil,
>
> What would a Summary object contain? I suspect it would contain a double
> value for each statistic (mean, var ...)? I assume a summary is read
> only?
Yes, a read-only "value bean" representing current values of statistics.
I get the feeling this is something that would contain various
> methods to return the statistical values. The ultimate debate arises,
> "Which statistics to place in it?"
The most commonly used ones -- those in the StatisticalSummary interface.
In such a case different users have
> different needs. If all we are talking about is an object that contains
> the current state of the SummaryStatisticsImpl, I'll argue this should
> be an exercise left up to the user.
I think that it would be useful to provide direct support for this, since
anyone who wants to pass statistical summaries around in an efficient way
will need this.
Phil
>
> Ken,
>
> I guess I am a little confused by the purpose of all this cloning? Are
> you attempting to maintain instantaneous statistics up to that point in
> the array? Are the copies in the statsArray ever incremented, or are
> they just replaced by new copies?
>
> I'm still interested in well established specs for cloning and
> serialization on these objects. We need to be able to persist and
> duplicate without any resulting errors.
>
> -Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [Math] Cloning statistics
Posted by "Mark R. Diggory" <md...@latte.harvard.edu>.
Phil Steitz wrote:
> Ken Geis wrote:
>
>> I'm playing with commons-math to implement a data mining algorithm and
>> I am having a performance problem.
>>
>> I am doing running statistics over an ordered set of data, storing the
>> statistics at each new value I come across. One way of doing this
>> would be to have an array of SummaryStatistics and do
>>
>> for (int i = 0; i < length; i++)
>> {
>> for (int j = i; j < length; j++)
>> {
>> statsArray[j].addValue(values[i]);
>> }
>> }
>>
>> another way is to do
>>
>> for (int i = 0; i < length; i++)
>> {
>> stats.addValue(values[i]);
>> statsArray[i] = SerializationUtils.clone(stats);
>> }
>>
>> A lot of these objects are marked Serializable, but clone methods do
>> not exist. That's why I use commons-lang SerializationUtils.
>> Unfortunately, that makes the cloning take up 50% of my runtime
>> because (de)serialization is expensive.
>>
>> I will probably patch the statistics classes, implementing enough of
>> clone() to make me happy. Would you like this patch?
>
>
> An efficient cloning method might be useful, but it would still carry
> around extra baggage and overhead for your use case
> (SummaryStatisticsImpl nests a bunch of little stats objects and other
> instance data).
>
> What we might want to do is add a StatisticalSummaryBean, implementing
> the StatisticalSummary interface and add a getSummary method to
> SummaryStatistics returning an instance of this "value bean" containing
> only the values of the statistics. Then you could just do
>
> for (int i = 0; i < length; i++)
> {
> stats.addValue(values[i]);
> statsArray[i] = stats.getSummary();
> }
>
> since I presume that all you will want to do with statsArray[i] is
> things like getMean(), getVariance(), etc. This would require much less
> overhead than cloning the whole SummaryStatisticsImpl instance each time.
>
> Since this would amount to a change to the SummaryStatistics interface,
> if we want to do it, we should do it now, before 1.0. I am +1 to this
> change and willing to implement it if no one objects.
>
> Phil
Phil,
What would a Summary object contain? I suspect it would contain a double
value for each statistic (mean, var ...)? I assume a summary is read
only? I get the feeling this is something that would contain various
methods to return the statistical values. The ultimate debate arises,
"Which statistics to place in it?" In such a case different users have
different needs. If all we are talking about is an object that contains
the current state of the SummaryStatisticsImpl, I'll argue this should
be an exercise left up to the user.
Ken,
I guess I am a little confused by the purpose of all this cloning? Are
you attempting to maintain instantaneous statistics up to that point in
the array? Are the copies in the statsArray ever incremented, or are
they just replaced by new copies?
I'm still interested in well established specs for cloning and
serialization on these objects. We need to be able to persist and
duplicate without any resulting errors.
-Mark
--
Mark Diggory
Software Developer
Harvard MIT Data Center
http://www.hmdc.harvard.edu
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [Math] Cloning statistics
Posted by Phil Steitz <ph...@steitz.com>.
Ken Geis wrote:
> I'm playing with commons-math to implement a data mining algorithm and I
> am having a performance problem.
>
> I am doing running statistics over an ordered set of data, storing the
> statistics at each new value I come across. One way of doing this would
> be to have an array of SummaryStatistics and do
>
> for (int i = 0; i < length; i++)
> {
> for (int j = i; j < length; j++)
> {
> statsArray[j].addValue(values[i]);
> }
> }
>
> another way is to do
>
> for (int i = 0; i < length; i++)
> {
> stats.addValue(values[i]);
> statsArray[i] = SerializationUtils.clone(stats);
> }
>
> A lot of these objects are marked Serializable, but clone methods do not
> exist. That's why I use commons-lang SerializationUtils. Unfortunately,
> that makes the cloning take up 50% of my runtime because
> (de)serialization is expensive.
>
> I will probably patch the statistics classes, implementing enough of
> clone() to make me happy. Would you like this patch?
An efficient cloning method might be useful, but it would still carry
around extra baggage and overhead for your use case (SummaryStatisticsImpl
nests a bunch of little stats objects and other instance data).
What we might want to do is add a StatisticalSummaryBean, implementing the
StatisticalSummary interface and add a getSummary method to
SummaryStatistics returning an instance of this "value bean" containing
only the values of the statistics. Then you could just do
for (int i = 0; i < length; i++)
{
stats.addValue(values[i]);
statsArray[i] = stats.getSummary();
}
since I presume that all you will want to do with statsArray[i] is things
like getMean(), getVariance(), etc. This would require much less overhead
than cloning the whole SummaryStatisticsImpl instance each time.
Since this would amount to a change to the SummaryStatistics interface, if
we want to do it, we should do it now, before 1.0. I am +1 to this change
and willing to implement it if no one objects.
Phil
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-dev-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [Math] Cloning statistics
Posted by "Mark R. Diggory" <md...@latte.harvard.edu>.
Serialization interfaces were added to support (de)serialization
(from)to file for persistence. Serialization actually uses the the
methods outlined in the Java Tutorial
http://java.sun.com/docs/books/tutorial/essential/io/providing.html
and is facilitated through the methods.
private void writeObject(ObjectOutputStream s) throws IOException;
private void readObject(ObjectInputStream s) throws IOException;
I believe this is quite independent of cloning. If I'm correct,
implementing the clone method on these classes shouldn't effect
serialization. But we should have consistent and comperable behavior for
both cloning and serialization
You know, we might also consider that the default clone method on
java.lang.Object returns a shallow copy. We should clarify the behavior
of our cloning, especially in the case where trying to copy a complex
network of objects.
I think we would be very interested in anything which would clarify the
cloning behavior of Math library.
-Mark
Ken Geis wrote:
> I'm playing with commons-math to implement a data mining algorithm and I
> am having a performance problem.
>
> I am doing running statistics over an ordered set of data, storing the
> statistics at each new value I come across. One way of doing this would
> be to have an array of SummaryStatistics and do
>
> for (int i = 0; i < length; i++)
> {
> for (int j = i; j < length; j++)
> {
> statsArray[j].addValue(values[i]);
> }
> }
>
> another way is to do
>
> for (int i = 0; i < length; i++)
> {
> stats.addValue(values[i]);
> statsArray[i] = SerializationUtils.clone(stats);
> }
>
> A lot of these objects are marked Serializable, but clone methods do not
> exist. That's why I use commons-lang SerializationUtils. Unfortunately,
> that makes the cloning take up 50% of my runtime because
> (de)serialization is expensive.
>
> I will probably patch the statistics classes, implementing enough of
> clone() to make me happy. Would you like this patch?
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-dev-help@jakarta.apache.org
>
--
Mark Diggory
Software Developer
Harvard MIT Data Center
http://www.hmdc.harvard.edu
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org