You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by "VanIngen, Erik (FIPS)" <Er...@fao.org> on 2010/09/29 12:13:01 UTC
[math]How to do standardizing (normalizing)
Hi Apache Commons Math users
I am looking for an easy way of standardizing my values a mean 0 and a standard deviation of 1. What is the best way to do that?
I have tried this:
DescriptiveStatistics stats = new DescriptiveStatistics();
// adding values
....
// Compute Mean and StandardDeviation
double mean = stats.getMean();
double std = stats.getStandardDeviation();
and then standardize each value according z = (x- mean)/std
But I would like to have just a function of standardize an array according the parameters mean and std. Is there something like this in Apache Math Commons?
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
RE: [math]How to do standardizing (normalizing)
Posted by Martin Gainty <mg...@hotmail.com>.
Good Morning Erik/Phil
I just encountered a bug in maven-surefire-api factory which went something like this
All array references such as double[], float[],int[] gacked the factory loader
reason:
this is not a Class that the factory is looking for
so what i did was for each situation where double[] double_array;
ArrayList<Double> double_array_list=new ArrayList();
for (int d=0;d<double.array.length(); d++)
{
Double double=new Double(double_array[i]);
double_array_list.add(double);
}
now my surefire factories (those that setup surefire testcases) will properly discover the unique class object called double_array_list
If anyone else has discovered this anomaly and has a solution please let me know
thanks,
Martin Gainty
______________________________________________
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> Date: Sat, 2 Oct 2010 20:50:49 -0400
> From: phil.steitz@gmail.com
> To: user@commons.apache.org
> Subject: Re: [math]How to do standardizing (normalizing)
>
> On 10/1/10 8:32 AM, VanIngen, Erik (FIPS) wrote:
> > Hi Luc and others,
> >
> > I have written the standardize function by myself (see below, including the tests). Would it be possible to have this added to Apache Math Commons?
> >
>
> Thanks for contributing!
>
> We should take discussion of this new feature to the dev list. It
> would be great if you could open a JIRA ticket and attach a patch
> including implementation code.
>
> We can talk about how to integrate this into [math] in JIRA comments
> and / or on the dev list. For now, I will just say that the
> simplest way to add this would be to add a static method called
> something like "normalize" to org.apache.commons.math.stat.StatUtils.
>
> See http://commons.apache.org/patches.html for info on how to create
> patches and attach them to JIRA tickets. Do not hesitate to ask
> either on dev list or in private emails if you need help getting set up.
>
> Thanks!
>
> Phil
>
> >
> >
> >
> >
> >
> > /**
> > * The standardise function does not seem to be in Apache math commons.
> > *
> > *
> > * @author Erik van Ingen
> > *
> > */
> > public class Standardize {
> >
> > /**
> > * Standardise the series, so in the end it is having mean of 0 and a standard deviation of 1.
> > *
> > *
> > * @param series
> > * @return
> > */
> > public static double[] run(double[] series) {
> > DescriptiveStatistics stats = new DescriptiveStatistics();
> >
> > // Add the data from the array
> > for (int i = 0; i< series.length; i++) {
> > stats.addValue(series[i]);
> > }
> >
> > // Compute mean and standard deviation
> > double currentMean = stats.getMean();
> > double currentstandardDeviation = stats.getStandardDeviation();
> >
> > // z = (x- mean)/standardDeviation
> > double[] newSeries = new double[series.length];
> >
> > for (int i = 0; i< series.length; i++) {
> > newSeries[i] = (series[i] - currentMean) / currentstandardDeviation;
> > }
> > return newSeries;
> > }
> >
> > }
> >
> >
> >
> > public class StandardizeTest {
> >
> > /**
> > * Run the test with the values 50 and 100 and assume standardized values with a dinstance of 0.01
> > */
> > @Test
> > public void testRun1() {
> > double series[] = { 50, 100 };
> > double expectedSeries[] = { -0.7, 0.7 };
> > double[] out = Standardize.run(series);
> > for (int i = 0; i< out.length; i++) {
> > assertEquals(out[i], expectedSeries[i], 0.01);
> > }
> >
> > }
> >
> > /**
> > * Run with 77 random values, assuming that the outcome has a mean of 0 and a standard deviation of 1.
> > *
> > *
> > *
> > */
> > @Test
> > public void testRun2() {
> > int length = 77;
> > double series[] = new double[length];
> >
> > for (int i = 0; i< length; i++) {
> > series[i] = Math.random();
> > }
> >
> > double standardizedSeries[] = Standardize.run(series);
> >
> > DescriptiveStatistics stats = new DescriptiveStatistics();
> >
> > // Add the data from the array
> > for (int i = 0; i< length; i++) {
> > stats.addValue(standardizedSeries[i]);
> > }
> >
> > double distance = 1E-10;
> > assertEquals(0.0, stats.getMean(), distance);
> > assertEquals(1.0, stats.getStandardDeviation(), distance);
> >
> > }
> >
> > }
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Luc Maisonobe [mailto:Luc.Maisonobe@free.fr]
> > Sent: 29 September 2010 18:54
> > To: Commons Users List
> > Subject: Re: [math]How to do standardizing (normalizing)
> >
> >
> > Le 29/09/2010 12:13, VanIngen, Erik (FIPS) a écrit :
> >> Hi Apache Commons Math users
> >>
> >> I am looking for an easy way of standardizing my values a mean 0 and a
> >> standard deviation of 1. What is the best way to do that?
> >>
> >> I have tried this:
> >> DescriptiveStatistics stats = new DescriptiveStatistics();
> >> // adding values
> >> ....
> >> // Compute Mean and StandardDeviation
> >> double mean = stats.getMean();
> >> double std = stats.getStandardDeviation();
> >>
> >> and then standardize each value according z = (x- mean)/std
> >>
> >> But I would like to have just a function of standardize an array
> >> according the parameters mean and std. Is there something like this in
> >> Apache Math Commons?
> >
> > I don't think we have such a function.
> >
> > Luc
> >
> >>
> >> Erik
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> >> For additional commands, e-mail: user-help@commons.apache.org
> >>
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> > For additional commands, e-mail: user-help@commons.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> > For additional commands, e-mail: user-help@commons.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
RE: [math]How to do standardizing (normalizing)
Posted by "VanIngen, Erik (FIPS)" <Er...@fao.org>.
Hi Phil,
I have created an issue
https://issues.apache.org/jira/browse/MATH-426
and added the code as a svn diff patch to StatUtils.
For 'Affected Version' I have noted 'Nightly Builds' which might not be the correct place.
Looking forward for the followup!
Cheers,
Erik van Ingen
-----Original Message-----
From: Phil Steitz [mailto:phil.steitz@gmail.com]
Sent: 03 October 2010 02:51
To: Commons Users List
Subject: Re: [math]How to do standardizing (normalizing)
On 10/1/10 8:32 AM, VanIngen, Erik (FIPS) wrote:
> Hi Luc and others,
>
> I have written the standardize function by myself (see below,
> including the tests). Would it be possible to have this added to
> Apache Math Commons?
>
Thanks for contributing!
We should take discussion of this new feature to the dev list. It
would be great if you could open a JIRA ticket and attach a patch
including implementation code.
We can talk about how to integrate this into [math] in JIRA comments
and / or on the dev list. For now, I will just say that the
simplest way to add this would be to add a static method called
something like "normalize" to org.apache.commons.math.stat.StatUtils.
See http://commons.apache.org/patches.html for info on how to create
patches and attach them to JIRA tickets. Do not hesitate to ask
either on dev list or in private emails if you need help getting set up.
Thanks!
Phil
>
>
>
>
>
> /**
> * The standardise function does not seem to be in Apache math commons.
> *
> *
> * @author Erik van Ingen
> *
> */
> public class Standardize {
>
> /**
> * Standardise the series, so in the end it is having mean of 0 and a standard deviation of 1.
> *
> *
> * @param series
> * @return
> */
> public static double[] run(double[] series) {
> DescriptiveStatistics stats = new
> DescriptiveStatistics();
>
> // Add the data from the array
> for (int i = 0; i< series.length; i++) {
> stats.addValue(series[i]);
> }
>
> // Compute mean and standard deviation
> double currentMean = stats.getMean();
> double currentstandardDeviation =
> stats.getStandardDeviation();
>
> // z = (x- mean)/standardDeviation
> double[] newSeries = new double[series.length];
>
> for (int i = 0; i< series.length; i++) {
> newSeries[i] = (series[i] - currentMean) / currentstandardDeviation;
> }
> return newSeries;
> }
>
> }
>
>
>
> public class StandardizeTest {
>
> /**
> * Run the test with the values 50 and 100 and assume standardized values with a dinstance of 0.01
> */
> @Test
> public void testRun1() {
> double series[] = { 50, 100 };
> double expectedSeries[] = { -0.7, 0.7 };
> double[] out = Standardize.run(series);
> for (int i = 0; i< out.length; i++) {
> assertEquals(out[i], expectedSeries[i], 0.01);
> }
>
> }
>
> /**
> * Run with 77 random values, assuming that the outcome has a mean of 0 and a standard deviation of 1.
> *
> *
> *
> */
> @Test
> public void testRun2() {
> int length = 77;
> double series[] = new double[length];
>
> for (int i = 0; i< length; i++) {
> series[i] = Math.random();
> }
>
> double standardizedSeries[] =
> Standardize.run(series);
>
> DescriptiveStatistics stats = new
> DescriptiveStatistics();
>
> // Add the data from the array
> for (int i = 0; i< length; i++) {
> stats.addValue(standardizedSeries[i]);
> }
>
> double distance = 1E-10;
> assertEquals(0.0, stats.getMean(), distance);
> assertEquals(1.0, stats.getStandardDeviation(),
> distance);
>
> }
>
> }
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: Luc Maisonobe [mailto:Luc.Maisonobe@free.fr]
> Sent: 29 September 2010 18:54
> To: Commons Users List
> Subject: Re: [math]How to do standardizing (normalizing)
>
>
> Le 29/09/2010 12:13, VanIngen, Erik (FIPS) a écrit :
>> Hi Apache Commons Math users
>>
>> I am looking for an easy way of standardizing my values a mean 0 and
>> a standard deviation of 1. What is the best way to do that?
>>
>> I have tried this:
>> DescriptiveStatistics stats = new DescriptiveStatistics(); // adding
>> values ....
>> // Compute Mean and StandardDeviation
>> double mean = stats.getMean();
>> double std = stats.getStandardDeviation();
>>
>> and then standardize each value according z = (x- mean)/std
>>
>> But I would like to have just a function of standardize an array
>> according the parameters mean and std. Is there something like this
>> in Apache Math Commons?
>
> I don't think we have such a function.
>
> Luc
>
>>
>> Erik
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>> For additional commands, e-mail: user-help@commons.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [math]How to do standardizing (normalizing)
Posted by Phil Steitz <ph...@gmail.com>.
On 10/1/10 8:32 AM, VanIngen, Erik (FIPS) wrote:
> Hi Luc and others,
>
> I have written the standardize function by myself (see below, including the tests). Would it be possible to have this added to Apache Math Commons?
>
Thanks for contributing!
We should take discussion of this new feature to the dev list. It
would be great if you could open a JIRA ticket and attach a patch
including implementation code.
We can talk about how to integrate this into [math] in JIRA comments
and / or on the dev list. For now, I will just say that the
simplest way to add this would be to add a static method called
something like "normalize" to org.apache.commons.math.stat.StatUtils.
See http://commons.apache.org/patches.html for info on how to create
patches and attach them to JIRA tickets. Do not hesitate to ask
either on dev list or in private emails if you need help getting set up.
Thanks!
Phil
>
>
>
>
>
> /**
> * The standardise function does not seem to be in Apache math commons.
> *
> *
> * @author Erik van Ingen
> *
> */
> public class Standardize {
>
> /**
> * Standardise the series, so in the end it is having mean of 0 and a standard deviation of 1.
> *
> *
> * @param series
> * @return
> */
> public static double[] run(double[] series) {
> DescriptiveStatistics stats = new DescriptiveStatistics();
>
> // Add the data from the array
> for (int i = 0; i< series.length; i++) {
> stats.addValue(series[i]);
> }
>
> // Compute mean and standard deviation
> double currentMean = stats.getMean();
> double currentstandardDeviation = stats.getStandardDeviation();
>
> // z = (x- mean)/standardDeviation
> double[] newSeries = new double[series.length];
>
> for (int i = 0; i< series.length; i++) {
> newSeries[i] = (series[i] - currentMean) / currentstandardDeviation;
> }
> return newSeries;
> }
>
> }
>
>
>
> public class StandardizeTest {
>
> /**
> * Run the test with the values 50 and 100 and assume standardized values with a dinstance of 0.01
> */
> @Test
> public void testRun1() {
> double series[] = { 50, 100 };
> double expectedSeries[] = { -0.7, 0.7 };
> double[] out = Standardize.run(series);
> for (int i = 0; i< out.length; i++) {
> assertEquals(out[i], expectedSeries[i], 0.01);
> }
>
> }
>
> /**
> * Run with 77 random values, assuming that the outcome has a mean of 0 and a standard deviation of 1.
> *
> *
> *
> */
> @Test
> public void testRun2() {
> int length = 77;
> double series[] = new double[length];
>
> for (int i = 0; i< length; i++) {
> series[i] = Math.random();
> }
>
> double standardizedSeries[] = Standardize.run(series);
>
> DescriptiveStatistics stats = new DescriptiveStatistics();
>
> // Add the data from the array
> for (int i = 0; i< length; i++) {
> stats.addValue(standardizedSeries[i]);
> }
>
> double distance = 1E-10;
> assertEquals(0.0, stats.getMean(), distance);
> assertEquals(1.0, stats.getStandardDeviation(), distance);
>
> }
>
> }
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: Luc Maisonobe [mailto:Luc.Maisonobe@free.fr]
> Sent: 29 September 2010 18:54
> To: Commons Users List
> Subject: Re: [math]How to do standardizing (normalizing)
>
>
> Le 29/09/2010 12:13, VanIngen, Erik (FIPS) a écrit :
>> Hi Apache Commons Math users
>>
>> I am looking for an easy way of standardizing my values a mean 0 and a
>> standard deviation of 1. What is the best way to do that?
>>
>> I have tried this:
>> DescriptiveStatistics stats = new DescriptiveStatistics();
>> // adding values
>> ....
>> // Compute Mean and StandardDeviation
>> double mean = stats.getMean();
>> double std = stats.getStandardDeviation();
>>
>> and then standardize each value according z = (x- mean)/std
>>
>> But I would like to have just a function of standardize an array
>> according the parameters mean and std. Is there something like this in
>> Apache Math Commons?
>
> I don't think we have such a function.
>
> Luc
>
>>
>> Erik
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>> For additional commands, e-mail: user-help@commons.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
RE: [math]How to do standardizing (normalizing)
Posted by "VanIngen, Erik (FIPS)" <Er...@fao.org>.
Hi Luc and others,
I have written the standardize function by myself (see below, including the tests). Would it be possible to have this added to Apache Math Commons?
/**
* The standardise function does not seem to be in Apache math commons.
*
*
* @author Erik van Ingen
*
*/
public class Standardize {
/**
* Standardise the series, so in the end it is having mean of 0 and a standard deviation of 1.
*
*
* @param series
* @return
*/
public static double[] run(double[] series) {
DescriptiveStatistics stats = new DescriptiveStatistics();
// Add the data from the array
for (int i = 0; i < series.length; i++) {
stats.addValue(series[i]);
}
// Compute mean and standard deviation
double currentMean = stats.getMean();
double currentstandardDeviation = stats.getStandardDeviation();
// z = (x- mean)/standardDeviation
double[] newSeries = new double[series.length];
for (int i = 0; i < series.length; i++) {
newSeries[i] = (series[i] - currentMean) / currentstandardDeviation;
}
return newSeries;
}
}
public class StandardizeTest {
/**
* Run the test with the values 50 and 100 and assume standardized values with a dinstance of 0.01
*/
@Test
public void testRun1() {
double series[] = { 50, 100 };
double expectedSeries[] = { -0.7, 0.7 };
double[] out = Standardize.run(series);
for (int i = 0; i < out.length; i++) {
assertEquals(out[i], expectedSeries[i], 0.01);
}
}
/**
* Run with 77 random values, assuming that the outcome has a mean of 0 and a standard deviation of 1.
*
*
*
*/
@Test
public void testRun2() {
int length = 77;
double series[] = new double[length];
for (int i = 0; i < length; i++) {
series[i] = Math.random();
}
double standardizedSeries[] = Standardize.run(series);
DescriptiveStatistics stats = new DescriptiveStatistics();
// Add the data from the array
for (int i = 0; i < length; i++) {
stats.addValue(standardizedSeries[i]);
}
double distance = 1E-10;
assertEquals(0.0, stats.getMean(), distance);
assertEquals(1.0, stats.getStandardDeviation(), distance);
}
}
-----Original Message-----
From: Luc Maisonobe [mailto:Luc.Maisonobe@free.fr]
Sent: 29 September 2010 18:54
To: Commons Users List
Subject: Re: [math]How to do standardizing (normalizing)
Le 29/09/2010 12:13, VanIngen, Erik (FIPS) a écrit :
> Hi Apache Commons Math users
>
> I am looking for an easy way of standardizing my values a mean 0 and a
> standard deviation of 1. What is the best way to do that?
>
> I have tried this:
> DescriptiveStatistics stats = new DescriptiveStatistics();
> // adding values
> ....
> // Compute Mean and StandardDeviation
> double mean = stats.getMean();
> double std = stats.getStandardDeviation();
>
> and then standardize each value according z = (x- mean)/std
>
> But I would like to have just a function of standardize an array
> according the parameters mean and std. Is there something like this in
> Apache Math Commons?
I don't think we have such a function.
Luc
>
> Erik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [math]How to do standardizing (normalizing)
Posted by Luc Maisonobe <Lu...@free.fr>.
Le 29/09/2010 12:13, VanIngen, Erik (FIPS) a écrit :
> Hi Apache Commons Math users
>
> I am looking for an easy way of standardizing my values a mean 0 and a standard deviation of 1. What is the best way to do that?
>
> I have tried this:
> DescriptiveStatistics stats = new DescriptiveStatistics();
> // adding values
> ....
> // Compute Mean and StandardDeviation
> double mean = stats.getMean();
> double std = stats.getStandardDeviation();
>
> and then standardize each value according z = (x- mean)/std
>
> But I would like to have just a function of standardize an array according the parameters mean and std. Is there something like this in Apache Math Commons?
I don't think we have such a function.
Luc
>
> Erik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org