You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-user@hadoop.apache.org by unmesha sreeveni <un...@gmail.com> on 2014/07/30 08:27:29 UTC

PCA in HAdoop MapReduce

I am little bit confused in doing PCA

I am trying it myown and I refered
http://nyx-www.informatik.uni-bremen.de/664/1/smith_tr_02.pdf

1.MR job to calculate mean
2.MR Job to substract mean from the input data(Data Adjust in .pdf)
3.MR job to find covarience and Calculated the eigenvectors and eigenvalues
of the covariance
matrix
4.---
Now need to transform my data
  Step 5: Deriving the new data set in pdf.
It is matrix multiplication ie MR3(step 3) * MR2(Step 2)

Matrix multiplication can be done in using a seperate mapreduce job but i
have MR2 data but I need to transpose MR3 data ie Final data =
RowFeatureVector(MR3) * Row DataAdjust(MR2)

WHile doing transpose in another MR,we will not be sure about the order of
elements they come from mapper

How will we achieve the above transpose.

Please correct me if I am wrong in PCA concepts.

-- 
*Thanks & Regards *


*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/

Re: PCA in HAdoop MapReduce

Posted by "Adaryl \"Bob\" Wakefield, MBA" <ad...@hotmail.com>.

This may not be the right place to ask this question. I asked a more generic question about how to do predictive modeling on hadoop and nobody answered. It perplexes me as well how to take these machine learning concepts and implement them in a Map Reduce paradigm.

Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics
913.938.6685
www.linkedin.com/in/bobwakefieldmba

From: unmesha sreeveni 
Sent: Wednesday, July 30, 2014 1:27 AM
To: User Hadoop 
Subject: PCA in HAdoop MapReduce


I am little bit confused in doing PCA 


I am trying it myown and I refered http://nyx-www.informatik.uni-bremen.de/664/1/smith_tr_02.pdf


1.MR job to calculate mean
2.MR Job to substract mean from the input data(Data Adjust in .pdf)
3.MR job to find covarience and Calculated the eigenvectors and eigenvalues of the covariance
matrix
4.---
Now need to transform my data
  Step 5: Deriving the new data set in pdf.
It is matrix multiplication ie MR3(step 3) * MR2(Step 2)


Matrix multiplication can be done in using a seperate mapreduce job but i have MR2 data but I need to transpose MR3 data ie Final data = RowFeatureVector(MR3) * Row DataAdjust(MR2)


WHile doing transpose in another MR,we will not be sure about the order of elements they come from mapper


How will we achieve the above transpose.


Please correct me if I am wrong in PCA concepts.

-- 

Thanks & Regards 


Unmesha Sreeveni U.B

Hadoop, Bigdata Developer
Center for Cyber Security | Amrita Vishwa Vidyapeetham

http://www.unmeshasreeveni.blogspot.in/

Re: PCA in HAdoop MapReduce

Posted by "Adaryl \"Bob\" Wakefield, MBA" <ad...@hotmail.com>.

This may not be the right place to ask this question. I asked a more generic question about how to do predictive modeling on hadoop and nobody answered. It perplexes me as well how to take these machine learning concepts and implement them in a Map Reduce paradigm.

Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics
913.938.6685
www.linkedin.com/in/bobwakefieldmba

From: unmesha sreeveni 
Sent: Wednesday, July 30, 2014 1:27 AM
To: User Hadoop 
Subject: PCA in HAdoop MapReduce


I am little bit confused in doing PCA 


I am trying it myown and I refered http://nyx-www.informatik.uni-bremen.de/664/1/smith_tr_02.pdf


1.MR job to calculate mean
2.MR Job to substract mean from the input data(Data Adjust in .pdf)
3.MR job to find covarience and Calculated the eigenvectors and eigenvalues of the covariance
matrix
4.---
Now need to transform my data
  Step 5: Deriving the new data set in pdf.
It is matrix multiplication ie MR3(step 3) * MR2(Step 2)


Matrix multiplication can be done in using a seperate mapreduce job but i have MR2 data but I need to transpose MR3 data ie Final data = RowFeatureVector(MR3) * Row DataAdjust(MR2)


WHile doing transpose in another MR,we will not be sure about the order of elements they come from mapper


How will we achieve the above transpose.


Please correct me if I am wrong in PCA concepts.

-- 

Thanks & Regards 


Unmesha Sreeveni U.B

Hadoop, Bigdata Developer
Center for Cyber Security | Amrita Vishwa Vidyapeetham

http://www.unmeshasreeveni.blogspot.in/

Re: PCA in HAdoop MapReduce

Posted by "Adaryl \"Bob\" Wakefield, MBA" <ad...@hotmail.com>.

This may not be the right place to ask this question. I asked a more generic question about how to do predictive modeling on hadoop and nobody answered. It perplexes me as well how to take these machine learning concepts and implement them in a Map Reduce paradigm.

Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics
913.938.6685
www.linkedin.com/in/bobwakefieldmba

From: unmesha sreeveni 
Sent: Wednesday, July 30, 2014 1:27 AM
To: User Hadoop 
Subject: PCA in HAdoop MapReduce


I am little bit confused in doing PCA 


I am trying it myown and I refered http://nyx-www.informatik.uni-bremen.de/664/1/smith_tr_02.pdf


1.MR job to calculate mean
2.MR Job to substract mean from the input data(Data Adjust in .pdf)
3.MR job to find covarience and Calculated the eigenvectors and eigenvalues of the covariance
matrix
4.---
Now need to transform my data
  Step 5: Deriving the new data set in pdf.
It is matrix multiplication ie MR3(step 3) * MR2(Step 2)


Matrix multiplication can be done in using a seperate mapreduce job but i have MR2 data but I need to transpose MR3 data ie Final data = RowFeatureVector(MR3) * Row DataAdjust(MR2)


WHile doing transpose in another MR,we will not be sure about the order of elements they come from mapper


How will we achieve the above transpose.


Please correct me if I am wrong in PCA concepts.

-- 

Thanks & Regards 


Unmesha Sreeveni U.B

Hadoop, Bigdata Developer
Center for Cyber Security | Amrita Vishwa Vidyapeetham

http://www.unmeshasreeveni.blogspot.in/

Re: PCA in HAdoop MapReduce

Posted by "Adaryl \"Bob\" Wakefield, MBA" <ad...@hotmail.com>.

This may not be the right place to ask this question. I asked a more generic question about how to do predictive modeling on hadoop and nobody answered. It perplexes me as well how to take these machine learning concepts and implement them in a Map Reduce paradigm.

Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics
913.938.6685
www.linkedin.com/in/bobwakefieldmba

From: unmesha sreeveni 
Sent: Wednesday, July 30, 2014 1:27 AM
To: User Hadoop 
Subject: PCA in HAdoop MapReduce


I am little bit confused in doing PCA 


I am trying it myown and I refered http://nyx-www.informatik.uni-bremen.de/664/1/smith_tr_02.pdf


1.MR job to calculate mean
2.MR Job to substract mean from the input data(Data Adjust in .pdf)
3.MR job to find covarience and Calculated the eigenvectors and eigenvalues of the covariance
matrix
4.---
Now need to transform my data
  Step 5: Deriving the new data set in pdf.
It is matrix multiplication ie MR3(step 3) * MR2(Step 2)


Matrix multiplication can be done in using a seperate mapreduce job but i have MR2 data but I need to transpose MR3 data ie Final data = RowFeatureVector(MR3) * Row DataAdjust(MR2)


WHile doing transpose in another MR,we will not be sure about the order of elements they come from mapper


How will we achieve the above transpose.


Please correct me if I am wrong in PCA concepts.

-- 

Thanks & Regards 


Unmesha Sreeveni U.B

Hadoop, Bigdata Developer
Center for Cyber Security | Amrita Vishwa Vidyapeetham

http://www.unmeshasreeveni.blogspot.in/