You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Sameer Tilak <ss...@live.com> on 2013/12/16 21:09:23 UTC
Data Vectorization
Hi All,
I have some questions regarding vectorization.
Here is my Pig script snippet.
AU = FOREACH A GENERATE myparser.myUDF(param1, param2); STORE AU into '/scratch/AU';
AU has the following format:
(userid, (item_view_history))
(27,(0,1,1,0,0))(28,(0,0,1,0,0))(29,(0,0,1,0,1))(30,(1,0,1,0,1))
I will have at least few hundred thousand numbers in the (item_view_history), for readability I am just showing 5 here.
I am not sure about how to get this data written to a format that Mahout's clustering algorithms will be able to parse. I have the following steps, but not sure if my understanding is correct. Any help with this will be great!
VectorizedInput = FOREACH AU GENERATE FLATTEN($0);
/*I am assuming the filed userid will be used as a key and will be written using $INT_CONVERTER', and the tuple will be written using $VECTOR_CONVERTER'. Is this correct?
STORE VectorizedInput into '/scratch/VectorizedInput' using $SEQFILE_STORAGE ('-c $INT_CONVERTER', '-c $VECTOR_CONVERTER');
RE: Data Vectorization
Posted by Sameer Tilak <ss...@live.com>.
Hi Suneel,Thanks for your reply. Here is the o/p again. Seems to have the same issue. I am running it off of trunk.
> Date: Mon, 16 Dec 2013 13:04:10 -0800
> From: suneel_marthi@yahoo.com
> Subject: Re: Data Vectorization
> To: user@mahout.apache.org
>
> Sorry I have not been following this thread but the output of seqdirectory seems suspect.á Are you working off of Mahout 0.8 or trunk?
> Could you try running seqdirectory again with '-xm sequential' option and repeat the subsequent steps?
> -bash-4.1$ ./mahout seqdirectory -xm sequential -i /scratch/VectorizedInput -o /scratch/VectorizedOutputSeqdir -c UTF-8 -chunk 5MAHOUT-JOB: /apps/mahout/trunk/examples/target/mahout-examples-0.9-SNAPSHOT-job.jar
13/12/16 13:07:16 INFO common.AbstractJob: Command line arguments: {--charset=[UTF-8], --chunkSize=[5], --endPhase=[2147483647], --fileFilterClass=[org.apache.mahout.text.PrefixAdditionFilter], --input=[/scratch/VectorizedInput], --keyPrefix=[], --method=[sequential], --output=[/scratch/VectorizedOutputSeqdir], --startPhase=[0], --tempDir=[temp]}13/12/16 13:07:16 INFO driver.MahoutDriver: Program took 990 ms (Minutes: 0.0165)
hadoop dfs -ls /scratch/VectorizedOutputSeqdirFound 1 items-rw-r--r-- 1 userid supergroup 633227 2013-12-16 13:07 /scratch/VectorizedOutputSeqdir/chunk-0
./mahout seq2sparse -i /scratch/VectorizedOutputSeqdir -o /scratch/VectorizedOutputSparce13/12/16 13:12:50 INFO common.HadoopUtil: Deleting /scratch/VectorizedOutputSparce/partial-vectors-013/12/16 13:12:50 INFO driver.MahoutDriver: Program took 313224 ms (Minutes: 5.2204)
hadoop dfs -ls /scratch/VectorizedOutputSparce
Found 7 itemsdrwxr-xr-x - userid supergroup 0 2013-12-16 13:10 /scratch/VectorizedOutputSparce/df-count-rw-r--r-- 1 userid supergroup 19765 2013-12-16 13:08 /scratch/VectorizedOutputSparce/dictionary.file-0-rw-r--r-- 1 userid supergroup 22953 2013-12-16 13:10 /scratch/VectorizedOutputSparce/frequency.file-0drwxr-xr-x - userid supergroup 0 2013-12-16 13:11 /scratch/VectorizedOutputSparce/tf-vectorsdrwxr-xr-x - userid supergroup 0 2013-12-16 13:12 /scratch/VectorizedOutputSparce/tfidf-vectorsdrwxr-xr-x - userid supergroup 0 2013-12-16 13:08 /scratch/VectorizedOutputSparce/tokenized-documentsdrwxr-xr-x - userid supergroup 0 2013-12-16 13:08 /scratch/VectorizedOutputSparce/wordcount
./mahout kmeans -i /scratch/VectorizedOutputSparce/tfidf-vectors/ -c /scratch/clusters -o /scratch/kmeans -x 10 -k 20 -ow13/12/16 13:18:18 INFO driver.MahoutDriver: Program took 43052 ms (Minutes: 0.7175333333333334)
-bash-4.1$ ./mahout clusterdump -i /scratch/kmeans/clusters-*-final -d /scratch/VectorizedOutputSparce/dictionary.file-0 -dt sequencefile -b 100 -n 20MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.Warning: $HADOOP_HOME is deprecated.
hadoop dfs -ls /scratch/kmeansFound 2 itemsdrwxr-xr-x - userid supergroup 0 2013-12-16 13:17 /scratch/kmeans/clusters-0drwxr-xr-x - userid supergroup 0 2013-12-16 13:18 /scratch/kmeans/clusters-1-final
MAHOUT-JOB: /apps/mahout/trunk/examples/target/mahout-examples-0.9-SNAPSHOT-job.jarWarning: $HADOOP_HOME is deprecated.
13/12/16 13:20:22 INFO common.AbstractJob: Command line arguments: {--dictionary=[/scratch/VectorizedOutputSparce/dictionary.file-0], --dictionaryType=[sequencefile], --distanceMeasure=[org.apache.mahout.common.distance.SquaredEuclideanDistanceMeasure], --endPhase=[2147483647], --input=[/scratch/kmeans/clusters-*-final], --numWords=[20], --outputFormat=[TEXT], --startPhase=[0], --substring=[100], --tempDir=[temp]}:VL-0{n=2 c=[0:13.950, 1:4.932, 110:3.186, 1107280:3.186, 1118:3.186, 1177:3.186, 120:2.612, 12065996 Top Terms: records => 23.841001510620117 output => 23.411399841308594 org => 19.115327835083008 bytes => 19.115327835083008 apache => 17.00798797607422 map => 15.120783805847168 counter => 14.424721717834473 counters => 14.247724533081055 hadoop => 14.189364433288574 0 => 13.949626922607422 input => 13.516578674316406 task_type => 12.743552207946777 taskid => 12.743552207946777 file_bytes_written => 12.743552207946777 filesystemcounters => 12.743552207946777 memory => 12.743552207946777 task => 12.338891983032227 _0_b => 11.036240577697754 _1_au => 11.036240577697754 _2_vectorizedinput => 11.036240577697754:VL-1{n=2 c=[0:6.975, 0.0.0.0:6.758, 0.05:3.186, 0.11.1:6.758, 0.9:7.124, 1:6.430, 1.0:3.186, 1.0.3:1 Top Terms: name => 59.7300910949707 property => 59.68759536743164 value => 59.68759536743164 jar => 17.303800582885742 users => 17.156532287597656 software => 17.00798797607422 1.0.3 => 16.858135223388672 lib => 16.625680923461914 libexec => 16.08794593811035 hadoop => 15.67484188079834 tmp => 15.607600212097168 userid => 14.545638084411621 true => 12.338891983032227 temp => 12.131500244140625 578898841 => 11.920500755310059 false => 10.56639575958252 institution => 10.323457717895508 usr => 9.81956672668457 openjdk => 9.557663917541504 1.7.0 => 9.288379669189453>
> Given that ur tf-vectors, tokenized documents etc are all zero bytes, I suspect your seqdirectory has failed.
>
>
>
>
> On Monday, December 16, 2013 3:58 PM, Sameer Tilak <ss...@live.com> wrote:
>
> It does not see to work :(.
> Here is who I use the generated sequence (described in my last email) file for clustering.
> ./mahout seqdirectory -i /scratch/VectorizedInput -o /scratch/VectorizedOutputSeqdir -c UTF-8 -chunk 513/12/12 11:33:53 INFO driver.MahoutDriver: Program took 24433 ms (Minutes: 0.40721666666666667)
> -bash-4.1$ hadoop dfs -lsá /scratch/VectorizedOutputSeqdirWarning: $HADOOP_HOME is deprecated.Found 3 items-rw-r--r--á 1 userid supergroupá á á á á 0 2013-12-12 11:33 /scratch/VectorizedOutputSeqdir/_SUCCESSdrwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:33 /scratch/VectorizedOutputSeqdir/_logs-rw-r--r--á 1 userid supergroupá á á 61940 2013-12-12 11:33 /scratch/VectorizedOutputSeqdir/part-m-00000
>
> ./mahout seq2sparse -iá /scratch/VectorizedOutputSeqdir -oá /scratch/VectorizedOutputSparceProgram took 361827 ms (Minutes: 6.03045)
> -bash-4.1$ hadoop dfs -lsá /scratch/VectorizedOutputSparceWarning: $HADOOP_HOME is deprecated.
> Found 7 itemsdrwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:38 /scratch/VectorizedOutputSparce/df-count-rw-r--r--á 1 userid supergroupá á á 19692 2013-12-12 11:36 /scratch/VectorizedOutputSparce/dictionary.file-0-rw-r--r--á 1 userid supergroupá á á 22893 2013-12-12 11:38 /scratch/VectorizedOutputSparce/frequency.file-0drwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:39 /scratch/VectorizedOutputSparce/tf-vectorsdrwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:41 /scratch/VectorizedOutputSparce/tfidf-vectorsdrwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:35 /scratch/VectorizedOutputSparce/tokenized-documentsdrwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:36 /scratch/VectorizedOutputSparce/wordcount
> ./mahout kmeans -iá /scratch/VectorizedOutputSparce/tfidf-vectors/ -c /scratch/clusters -o /scratch/kmeans -x 10 -k 20 -owProgram took 43559 ms (Minutes: 0.7259833333333333)
> -bash-4.1$ hadoop dfs -ls /scratch/kmeansWarning: $HADOOP_HOME is deprecated.Found 2 itemsdrwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:57 /scratch/kmeans/clusters-0drwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:58 /scratch/kmeans/clusters-1-final
>
> ./mahout clusterdump -i /scratch/kmeans/clusters-*-final -dá /scratch/VectorizedOutputSparce/dictionary.file-0 -dt sequencefile -b 100 -n 2013/12/12 12:28:43 INFO clustering.ClusterDumper: Wrote 7 clusters13/12/12 12:28:43 INFO driver.MahoutDriver: Program took 1066 ms (Minutes: 0.017766666666666667)
>
> Running on hadoop, using /users/p529444/software/hadoop-1.0.3/bin/hadoop and HADOOP_CONF_DIR=/apps/hadoop/hadoop-confMAHOUT-JOB: /apps/mahout/trunk/examples/target/mahout-examples-0.9-SNAPSHOT-job.jarWarning: $HADOOP_HOME is deprecated.
> 13/12/16 12:52:49 INFO common.AbstractJob: Command line arguments: {--dictionary=[/scratch/VectorizedOutputSparce/dictionary.file-0], --dictionaryType=[sequencefile], --distanceMeasure=[org.apache.mahout.common.distance.SquaredEuclideanDistanceMeasure], --endPhase=[2147483647], --input=[/scratch/kmeans/clusters-*-final], --numWords=[20], --outputFormat=[TEXT], --startPhase=[0], --substring=[100], --tempDir=[temp]}:VL-0{n=2 c=[0:13.950, 1:4.932, 110:3.186, 1107280:3.186, 1118:3.186, 1177:3.186, 120:2.612, 12065996ááá Top Terms:ááá ááá recordsá á á á á á á á á á á á á á á á =>á 23.841001510620117ááá ááá outputá á á á á á á á á á á á á á á á á =>á 23.411399841308594ááá ááá orgá á á á á á á á á á á á á á á á á á =>á 19.115327835083008ááá ááá bytesá á á á á á á á á á á á á á á á á =>á 19.115327835083008ááá ááá apacheá á á á á á á á á á á á
> á á á á á =>á 17.00798797607422ááá ááá mapá á á á á á á á á á á á á á á á á á =>á 15.120783805847168ááá ááá counterá á á á á á á á á á á á á á á á =>á 14.424721717834473ááá ááá countersá á á á á á á á á á á á á á á á =>á 14.247724533081055ááá ááá hadoopá á á á á á á á á á á á á á á á á =>á 14.189364433288574ááá ááá 0á á á á á á á á á á á á á á á á á á á =>á 13.949626922607422ááá ááá inputá á á á á á á á á á á á á á á á á =>á 13.516578674316406ááá ááá task_typeá á á á á á á á á á á á á á á =>á 12.743552207946777ááá ááá taskidá á á á á á á á á á á á á á á á á =>á 12.743552207946777ááá ááá file_bytes_writtená á á á á á á á á á á =>á 12.743552207946777ááá ááá filesystemcountersá á á á á á á á á á á =>á
> 12.743552207946777ááá ááá memoryá á á á á á á á á á á á á á á á á =>á 12.743552207946777ááá ááá taská á á á á á á á á á á á á á á á á á =>á 12.338891983032227ááá ááá _0_bá á á á á á á á á á á á á á á á á á =>á 11.036240577697754ááá ááá _1_auá á á á á á á á á á á á á á á á á =>á 11.036240577697754ááá ááá _2_vectorizedinputá á á á á á á á á á á =>á 11.036240577697754:VL-1{n=2 c=[0:6.975, 0.0.0.0:6.758, 0.05:3.186, 0.11.1:6.758, 0.9:7.124, 1:6.430, 1.0:3.186, 1.0.3:1ááá Top Terms:ááá ááá nameá á á á á á á á á á á á á á á á á á =>á á 59.7300910949707ááá ááá propertyá á á á á á á á á á á á á á á á =>á 59.68759536743164ááá ááá valueá á á á á á á á á á á á á á á á á =>á 59.68759536743164ááá ááá jará á á á á á á á á á á á á á á á á á =>á
> 17.303800582885742ááá ááá usersá á á á á á á á á á á á á á á á á =>á 17.156532287597656ááá ááá softwareá á á á á á á á á á á á á á á á =>á 17.00798797607422ááá ááá 1.0.3á á á á á á á á á á á á á á á á á =>á 16.858135223388672ááá ááá libá á á á á á á á á á á á á á á á á á =>á 16.625680923461914ááá ááá libexecá á á á á á á á á á á á á á á á =>á 16.08794593811035ááá ááá hadoopá á á á á á á á á á á á á á á á á =>á 15.67484188079834ááá ááá tmpá á á á á á á á á á á á á á á á á á =>á 15.607600212097168ááá ááá useridá á á á á á á á á á á á á á á á =>á 14.545638084411621ááá ááá trueá á á á á á á á á á á á á á á á á á =>á 12.338891983032227ááá ááá tempá á á á á á á á á á á á á á á á á á =>á 12.131500244140625ááá
> ááá 578898841á á á á á á á á á á á á á á á =>á 11.920500755310059ááá ááá falseá á á á á á á á á á á á á á á á á =>á 10.56639575958252ááá ááá institutioná á á á á á á á á á á á =>á 10.323457717895508ááá ááá usrá á á á á á á á á á á á á á á á á á =>á á 9.81956672668457ááá ááá openjdká á á á á á á á á á á á á á á á =>á 9.557663917541504ááá ááá 1.7.0á á á á á á á á á á á á á á á á á =>á 9.288379669189453
>
>
> > Date: Mon, 16 Dec 2013 12:13:52 -0800
> > Subject: Re: Data Vectorization
> > From: andrew.musselman@gmail.com
> > To: user@mahout.apache.org
> >
> > Looks reasonable.á Does it work?
> >
> >
> > On Mon, Dec 16, 2013 at 12:09 PM, Sameer Tilak <ss...@live.com> wrote:
> >
> > > Hi All,
> > > I have some questions regarding vectorization.
> > >
> > > Here is my Pig script snippet.
> > >
> > > AU = FOREACH A GENERATE myparser.myUDF(param1, param2); STORE AU into
> > > '/scratch/AU';
> > > AU has the following format:
> > > (userid, (item_view_history))
> > > (27,(0,1,1,0,0))(28,(0,0,1,0,0))(29,(0,0,1,0,1))(30,(1,0,1,0,1))
> > > I will have at least few hundred thousand numbers in the
> > >á (item_view_history), for readability I am just showing 5 here.
> > > I am not sure about how to get this data written to a format that Mahout's
> > > clustering algorithms will be able to parse. I have the following steps,
> > > but not sure if my understanding is correct. Any help with this will be
> > > great!
> > >
> > > VectorizedInput = FOREACH AU GENERATE FLATTEN($0);
> > >
> > > /*I am assuming the filed userid will be used as a key and will be written
> > > using $INT_CONVERTER', and the tuple will be written using
> > > $VECTOR_CONVERTER'. Is this correct?
> > > STORE VectorizedInput into '/scratch/VectorizedInput' using
> > > $SEQFILE_STORAGE ('-c $INT_CONVERTER', '-c $VECTOR_CONVERTER');
> > >
RE: Data Vectorization
Posted by Sameer Tilak <ss...@live.com>.
BTW,
The i/p (/scratch/VectorizedInput) to the ./mahout seqdirectory was generated using STORE VectorizedInput into '/scratch/VectorizedInput' using $SEQFILE_STORAGE ('-c $INT_CONVERTER', '-c $VECTOR_CONVERTER') and had the following contents:
Found 7 items-rw-r--r-- 1 userid supergroup 0 2013-12-16 12:43 /scratch/VectorizedInput/_SUCCESSdrwxr-xr-x - userid supergroup 0 2013-12-16 12:41 /scratch/VectorizedInput/_logs-rw-r--r-- 1 userid supergroup 102003 2013-12-16 12:42 /scratch/VectorizedInput/part-m-00000-rw-r--r-- 1 userid supergroup 101011 2013-12-16 12:42 /scratch/VectorizedInput/part-m-00001-rw-r--r-- 1 userid supergroup 105641 2013-12-16 12:42 /scratch/VectorizedInput/part-m-00002-rw-r--r-- 1 userid supergroup 107659 2013-12-16 12:42 /scratch/VectorizedInput/part-m-00003-rw-r--r-- 1 userid supergroup 106525 2013-12-16 12:42 /scratch/VectorizedInput/part-m-00004
> Date: Mon, 16 Dec 2013 13:04:10 -0800
> From: suneel_marthi@yahoo.com
> Subject: Re: Data Vectorization
> To: user@mahout.apache.org
>
> Sorry I have not been following this thread but the output of seqdirectory seems suspect.á Are you working off of Mahout 0.8 or trunk?
> Could you try running seqdirectory again with '-xm sequential' option and repeat the subsequent steps?
>
>
> Given that ur tf-vectors, tokenized documents etc are all zero bytes, I suspect your seqdirectory has failed.
>
>
>
>
> On Monday, December 16, 2013 3:58 PM, Sameer Tilak <ss...@live.com> wrote:
>
> It does not see to work :(.
> Here is who I use the generated sequence (described in my last email) file for clustering.
> ./mahout seqdirectory -i /scratch/VectorizedInput -o /scratch/VectorizedOutputSeqdir -c UTF-8 -chunk 513/12/12 11:33:53 INFO driver.MahoutDriver: Program took 24433 ms (Minutes: 0.40721666666666667)
> -bash-4.1$ hadoop dfs -lsá /scratch/VectorizedOutputSeqdirWarning: $HADOOP_HOME is deprecated.Found 3 items-rw-r--r--á 1 userid supergroupá á á á á 0 2013-12-12 11:33 /scratch/VectorizedOutputSeqdir/_SUCCESSdrwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:33 /scratch/VectorizedOutputSeqdir/_logs-rw-r--r--á 1 userid supergroupá á á 61940 2013-12-12 11:33 /scratch/VectorizedOutputSeqdir/part-m-00000
>
> ./mahout seq2sparse -iá /scratch/VectorizedOutputSeqdir -oá /scratch/VectorizedOutputSparceProgram took 361827 ms (Minutes: 6.03045)
> -bash-4.1$ hadoop dfs -lsá /scratch/VectorizedOutputSparceWarning: $HADOOP_HOME is deprecated.
> Found 7 itemsdrwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:38 /scratch/VectorizedOutputSparce/df-count-rw-r--r--á 1 userid supergroupá á á 19692 2013-12-12 11:36 /scratch/VectorizedOutputSparce/dictionary.file-0-rw-r--r--á 1 userid supergroupá á á 22893 2013-12-12 11:38 /scratch/VectorizedOutputSparce/frequency.file-0drwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:39 /scratch/VectorizedOutputSparce/tf-vectorsdrwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:41 /scratch/VectorizedOutputSparce/tfidf-vectorsdrwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:35 /scratch/VectorizedOutputSparce/tokenized-documentsdrwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:36 /scratch/VectorizedOutputSparce/wordcount
> ./mahout kmeans -iá /scratch/VectorizedOutputSparce/tfidf-vectors/ -c /scratch/clusters -o /scratch/kmeans -x 10 -k 20 -owProgram took 43559 ms (Minutes: 0.7259833333333333)
> -bash-4.1$ hadoop dfs -ls /scratch/kmeansWarning: $HADOOP_HOME is deprecated.Found 2 itemsdrwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:57 /scratch/kmeans/clusters-0drwxr-xr-xá - userid supergroupá á á á á 0 2013-12-12 11:58 /scratch/kmeans/clusters-1-final
>
> ./mahout clusterdump -i /scratch/kmeans/clusters-*-final -dá /scratch/VectorizedOutputSparce/dictionary.file-0 -dt sequencefile -b 100 -n 2013/12/12 12:28:43 INFO clustering.ClusterDumper: Wrote 7 clusters13/12/12 12:28:43 INFO driver.MahoutDriver: Program took 1066 ms (Minutes: 0.017766666666666667)
>
> Running on hadoop, using /users/p529444/software/hadoop-1.0.3/bin/hadoop and HADOOP_CONF_DIR=/apps/hadoop/hadoop-confMAHOUT-JOB: /apps/mahout/trunk/examples/target/mahout-examples-0.9-SNAPSHOT-job.jarWarning: $HADOOP_HOME is deprecated.
> 13/12/16 12:52:49 INFO common.AbstractJob: Command line arguments: {--dictionary=[/scratch/VectorizedOutputSparce/dictionary.file-0], --dictionaryType=[sequencefile], --distanceMeasure=[org.apache.mahout.common.distance.SquaredEuclideanDistanceMeasure], --endPhase=[2147483647], --input=[/scratch/kmeans/clusters-*-final], --numWords=[20], --outputFormat=[TEXT], --startPhase=[0], --substring=[100], --tempDir=[temp]}:VL-0{n=2 c=[0:13.950, 1:4.932, 110:3.186, 1107280:3.186, 1118:3.186, 1177:3.186, 120:2.612, 12065996ááá Top Terms:ááá ááá recordsá á á á á á á á á á á á á á á á =>á 23.841001510620117ááá ááá outputá á á á á á á á á á á á á á á á á =>á 23.411399841308594ááá ááá orgá á á á á á á á á á á á á á á á á á =>á 19.115327835083008ááá ááá bytesá á á á á á á á á á á á á á á á á =>á 19.115327835083008ááá ááá apacheá á á á á á á á á á á á
> á á á á á =>á 17.00798797607422ááá ááá mapá á á á á á á á á á á á á á á á á á =>á 15.120783805847168ááá ááá counterá á á á á á á á á á á á á á á á =>á 14.424721717834473ááá ááá countersá á á á á á á á á á á á á á á á =>á 14.247724533081055ááá ááá hadoopá á á á á á á á á á á á á á á á á =>á 14.189364433288574ááá ááá 0á á á á á á á á á á á á á á á á á á á =>á 13.949626922607422ááá ááá inputá á á á á á á á á á á á á á á á á =>á 13.516578674316406ááá ááá task_typeá á á á á á á á á á á á á á á =>á 12.743552207946777ááá ááá taskidá á á á á á á á á á á á á á á á á =>á 12.743552207946777ááá ááá file_bytes_writtená á á á á á á á á á á =>á 12.743552207946777ááá ááá filesystemcountersá á á á á á á á á á á =>á
> 12.743552207946777ááá ááá memoryá á á á á á á á á á á á á á á á á =>á 12.743552207946777ááá ááá taská á á á á á á á á á á á á á á á á á =>á 12.338891983032227ááá ááá _0_bá á á á á á á á á á á á á á á á á á =>á 11.036240577697754ááá ááá _1_auá á á á á á á á á á á á á á á á á =>á 11.036240577697754ááá ááá _2_vectorizedinputá á á á á á á á á á á =>á 11.036240577697754:VL-1{n=2 c=[0:6.975, 0.0.0.0:6.758, 0.05:3.186, 0.11.1:6.758, 0.9:7.124, 1:6.430, 1.0:3.186, 1.0.3:1ááá Top Terms:ááá ááá nameá á á á á á á á á á á á á á á á á á =>á á 59.7300910949707ááá ááá propertyá á á á á á á á á á á á á á á á =>á 59.68759536743164ááá ááá valueá á á á á á á á á á á á á á á á á =>á 59.68759536743164ááá ááá jará á á á á á á á á á á á á á á á á á =>á
> 17.303800582885742ááá ááá usersá á á á á á á á á á á á á á á á á =>á 17.156532287597656ááá ááá softwareá á á á á á á á á á á á á á á á =>á 17.00798797607422ááá ááá 1.0.3á á á á á á á á á á á á á á á á á =>á 16.858135223388672ááá ááá libá á á á á á á á á á á á á á á á á á =>á 16.625680923461914ááá ááá libexecá á á á á á á á á á á á á á á á =>á 16.08794593811035ááá ááá hadoopá á á á á á á á á á á á á á á á á =>á 15.67484188079834ááá ááá tmpá á á á á á á á á á á á á á á á á á =>á 15.607600212097168ááá ááá useridá á á á á á á á á á á á á á á á =>á 14.545638084411621ááá ááá trueá á á á á á á á á á á á á á á á á á =>á 12.338891983032227ááá ááá tempá á á á á á á á á á á á á á á á á á =>á 12.131500244140625ááá
> ááá 578898841á á á á á á á á á á á á á á á =>á 11.920500755310059ááá ááá falseá á á á á á á á á á á á á á á á á =>á 10.56639575958252ááá ááá institutioná á á á á á á á á á á á =>á 10.323457717895508ááá ááá usrá á á á á á á á á á á á á á á á á á =>á á 9.81956672668457ááá ááá openjdká á á á á á á á á á á á á á á á =>á 9.557663917541504ááá ááá 1.7.0á á á á á á á á á á á á á á á á á =>á 9.288379669189453
>
>
> > Date: Mon, 16 Dec 2013 12:13:52 -0800
> > Subject: Re: Data Vectorization
> > From: andrew.musselman@gmail.com
> > To: user@mahout.apache.org
> >
> > Looks reasonable.á Does it work?
> >
> >
> > On Mon, Dec 16, 2013 at 12:09 PM, Sameer Tilak <ss...@live.com> wrote:
> >
> > > Hi All,
> > > I have some questions regarding vectorization.
> > >
> > > Here is my Pig script snippet.
> > >
> > > AU = FOREACH A GENERATE myparser.myUDF(param1, param2); STORE AU into
> > > '/scratch/AU';
> > > AU has the following format:
> > > (userid, (item_view_history))
> > > (27,(0,1,1,0,0))(28,(0,0,1,0,0))(29,(0,0,1,0,1))(30,(1,0,1,0,1))
> > > I will have at least few hundred thousand numbers in the
> > >á (item_view_history), for readability I am just showing 5 here.
> > > I am not sure about how to get this data written to a format that Mahout's
> > > clustering algorithms will be able to parse. I have the following steps,
> > > but not sure if my understanding is correct. Any help with this will be
> > > great!
> > >
> > > VectorizedInput = FOREACH AU GENERATE FLATTEN($0);
> > >
> > > /*I am assuming the filed userid will be used as a key and will be written
> > > using $INT_CONVERTER', and the tuple will be written using
> > > $VECTOR_CONVERTER'. Is this correct?
> > > STORE VectorizedInput into '/scratch/VectorizedInput' using
> > > $SEQFILE_STORAGE ('-c $INT_CONVERTER', '-c $VECTOR_CONVERTER');
> > >
Re: Data Vectorization
Posted by Suneel Marthi <su...@yahoo.com>.
Sorry I have not been following this thread but the output of seqdirectory seems suspect. Are you working off of Mahout 0.8 or trunk?
Could you try running seqdirectory again with '-xm sequential' option and repeat the subsequent steps?
Given that ur tf-vectors, tokenized documents etc are all zero bytes, I suspect your seqdirectory has failed.
On Monday, December 16, 2013 3:58 PM, Sameer Tilak <ss...@live.com> wrote:
It does not see to work :(.
Here is who I use the generated sequence (described in my last email) file for clustering.
./mahout seqdirectory -i /scratch/VectorizedInput -o /scratch/VectorizedOutputSeqdir -c UTF-8 -chunk 513/12/12 11:33:53 INFO driver.MahoutDriver: Program took 24433 ms (Minutes: 0.40721666666666667)
-bash-4.1$ hadoop dfs -ls /scratch/VectorizedOutputSeqdirWarning: $HADOOP_HOME is deprecated.Found 3 items-rw-r--r-- 1 userid supergroup 0 2013-12-12 11:33 /scratch/VectorizedOutputSeqdir/_SUCCESSdrwxr-xr-x - userid supergroup 0 2013-12-12 11:33 /scratch/VectorizedOutputSeqdir/_logs-rw-r--r-- 1 userid supergroup 61940 2013-12-12 11:33 /scratch/VectorizedOutputSeqdir/part-m-00000
./mahout seq2sparse -i /scratch/VectorizedOutputSeqdir -o /scratch/VectorizedOutputSparceProgram took 361827 ms (Minutes: 6.03045)
-bash-4.1$ hadoop dfs -ls /scratch/VectorizedOutputSparceWarning: $HADOOP_HOME is deprecated.
Found 7 itemsdrwxr-xr-x - userid supergroup 0 2013-12-12 11:38 /scratch/VectorizedOutputSparce/df-count-rw-r--r-- 1 userid supergroup 19692 2013-12-12 11:36 /scratch/VectorizedOutputSparce/dictionary.file-0-rw-r--r-- 1 userid supergroup 22893 2013-12-12 11:38 /scratch/VectorizedOutputSparce/frequency.file-0drwxr-xr-x - userid supergroup 0 2013-12-12 11:39 /scratch/VectorizedOutputSparce/tf-vectorsdrwxr-xr-x - userid supergroup 0 2013-12-12 11:41 /scratch/VectorizedOutputSparce/tfidf-vectorsdrwxr-xr-x - userid supergroup 0 2013-12-12 11:35 /scratch/VectorizedOutputSparce/tokenized-documentsdrwxr-xr-x - userid supergroup 0 2013-12-12 11:36 /scratch/VectorizedOutputSparce/wordcount
./mahout kmeans -i /scratch/VectorizedOutputSparce/tfidf-vectors/ -c /scratch/clusters -o /scratch/kmeans -x 10 -k 20 -owProgram took 43559 ms (Minutes: 0.7259833333333333)
-bash-4.1$ hadoop dfs -ls /scratch/kmeansWarning: $HADOOP_HOME is deprecated.Found 2 itemsdrwxr-xr-x - userid supergroup 0 2013-12-12 11:57 /scratch/kmeans/clusters-0drwxr-xr-x - userid supergroup 0 2013-12-12 11:58 /scratch/kmeans/clusters-1-final
./mahout clusterdump -i /scratch/kmeans/clusters-*-final -d /scratch/VectorizedOutputSparce/dictionary.file-0 -dt sequencefile -b 100 -n 2013/12/12 12:28:43 INFO clustering.ClusterDumper: Wrote 7 clusters13/12/12 12:28:43 INFO driver.MahoutDriver: Program took 1066 ms (Minutes: 0.017766666666666667)
Running on hadoop, using /users/p529444/software/hadoop-1.0.3/bin/hadoop and HADOOP_CONF_DIR=/apps/hadoop/hadoop-confMAHOUT-JOB: /apps/mahout/trunk/examples/target/mahout-examples-0.9-SNAPSHOT-job.jarWarning: $HADOOP_HOME is deprecated.
13/12/16 12:52:49 INFO common.AbstractJob: Command line arguments: {--dictionary=[/scratch/VectorizedOutputSparce/dictionary.file-0], --dictionaryType=[sequencefile], --distanceMeasure=[org.apache.mahout.common.distance.SquaredEuclideanDistanceMeasure], --endPhase=[2147483647], --input=[/scratch/kmeans/clusters-*-final], --numWords=[20], --outputFormat=[TEXT], --startPhase=[0], --substring=[100], --tempDir=[temp]}:VL-0{n=2 c=[0:13.950, 1:4.932, 110:3.186, 1107280:3.186, 1118:3.186, 1177:3.186, 120:2.612, 12065996 Top Terms: records => 23.841001510620117 output => 23.411399841308594 org => 19.115327835083008 bytes => 19.115327835083008 apache
=> 17.00798797607422 map => 15.120783805847168 counter => 14.424721717834473 counters => 14.247724533081055 hadoop => 14.189364433288574 0 => 13.949626922607422 input => 13.516578674316406 task_type => 12.743552207946777 taskid => 12.743552207946777 file_bytes_written => 12.743552207946777 filesystemcounters =>
12.743552207946777 memory => 12.743552207946777 task => 12.338891983032227 _0_b => 11.036240577697754 _1_au => 11.036240577697754 _2_vectorizedinput => 11.036240577697754:VL-1{n=2 c=[0:6.975, 0.0.0.0:6.758, 0.05:3.186, 0.11.1:6.758, 0.9:7.124, 1:6.430, 1.0:3.186, 1.0.3:1 Top Terms: name => 59.7300910949707 property => 59.68759536743164 value => 59.68759536743164 jar =>
17.303800582885742 users => 17.156532287597656 software => 17.00798797607422 1.0.3 => 16.858135223388672 lib => 16.625680923461914 libexec => 16.08794593811035 hadoop => 15.67484188079834 tmp => 15.607600212097168 userid => 14.545638084411621 true => 12.338891983032227 temp => 12.131500244140625
578898841 => 11.920500755310059 false => 10.56639575958252 institution => 10.323457717895508 usr => 9.81956672668457 openjdk => 9.557663917541504 1.7.0 => 9.288379669189453
> Date: Mon, 16 Dec 2013 12:13:52 -0800
> Subject: Re: Data Vectorization
> From: andrew.musselman@gmail.com
> To: user@mahout.apache.org
>
> Looks reasonable. Does it work?
>
>
> On Mon, Dec 16, 2013 at 12:09 PM, Sameer Tilak <ss...@live.com> wrote:
>
> > Hi All,
> > I have some questions regarding vectorization.
> >
> > Here is my Pig script snippet.
> >
> > AU = FOREACH A GENERATE myparser.myUDF(param1, param2); STORE AU into
> > '/scratch/AU';
> > AU has the following format:
> > (userid, (item_view_history))
> > (27,(0,1,1,0,0))(28,(0,0,1,0,0))(29,(0,0,1,0,1))(30,(1,0,1,0,1))
> > I will have at least few hundred thousand numbers in the
> > (item_view_history), for readability I am just showing 5 here.
> > I am not sure about how to get this data written to a format that Mahout's
> > clustering algorithms will be able to parse. I have the following steps,
> > but not sure if my understanding is correct. Any help with this will be
> > great!
> >
> > VectorizedInput = FOREACH AU GENERATE FLATTEN($0);
> >
> > /*I am assuming the filed userid will be used as a key and will be written
> > using $INT_CONVERTER', and the tuple will be written using
> > $VECTOR_CONVERTER'. Is this correct?
> > STORE VectorizedInput into '/scratch/VectorizedInput' using
> > $SEQFILE_STORAGE ('-c $INT_CONVERTER', '-c $VECTOR_CONVERTER');
> >
RE: Data Vectorization
Posted by Sameer Tilak <ss...@live.com>.
It does not see to work :(.
Here is who I use the generated sequence (described in my last email) file for clustering.
./mahout seqdirectory -i /scratch/VectorizedInput -o /scratch/VectorizedOutputSeqdir -c UTF-8 -chunk 513/12/12 11:33:53 INFO driver.MahoutDriver: Program took 24433 ms (Minutes: 0.40721666666666667)
-bash-4.1$ hadoop dfs -ls /scratch/VectorizedOutputSeqdirWarning: $HADOOP_HOME is deprecated.Found 3 items-rw-r--r-- 1 userid supergroup 0 2013-12-12 11:33 /scratch/VectorizedOutputSeqdir/_SUCCESSdrwxr-xr-x - userid supergroup 0 2013-12-12 11:33 /scratch/VectorizedOutputSeqdir/_logs-rw-r--r-- 1 userid supergroup 61940 2013-12-12 11:33 /scratch/VectorizedOutputSeqdir/part-m-00000
./mahout seq2sparse -i /scratch/VectorizedOutputSeqdir -o /scratch/VectorizedOutputSparceProgram took 361827 ms (Minutes: 6.03045)
-bash-4.1$ hadoop dfs -ls /scratch/VectorizedOutputSparceWarning: $HADOOP_HOME is deprecated.
Found 7 itemsdrwxr-xr-x - userid supergroup 0 2013-12-12 11:38 /scratch/VectorizedOutputSparce/df-count-rw-r--r-- 1 userid supergroup 19692 2013-12-12 11:36 /scratch/VectorizedOutputSparce/dictionary.file-0-rw-r--r-- 1 userid supergroup 22893 2013-12-12 11:38 /scratch/VectorizedOutputSparce/frequency.file-0drwxr-xr-x - userid supergroup 0 2013-12-12 11:39 /scratch/VectorizedOutputSparce/tf-vectorsdrwxr-xr-x - userid supergroup 0 2013-12-12 11:41 /scratch/VectorizedOutputSparce/tfidf-vectorsdrwxr-xr-x - userid supergroup 0 2013-12-12 11:35 /scratch/VectorizedOutputSparce/tokenized-documentsdrwxr-xr-x - userid supergroup 0 2013-12-12 11:36 /scratch/VectorizedOutputSparce/wordcount
./mahout kmeans -i /scratch/VectorizedOutputSparce/tfidf-vectors/ -c /scratch/clusters -o /scratch/kmeans -x 10 -k 20 -owProgram took 43559 ms (Minutes: 0.7259833333333333)
-bash-4.1$ hadoop dfs -ls /scratch/kmeansWarning: $HADOOP_HOME is deprecated.Found 2 itemsdrwxr-xr-x - userid supergroup 0 2013-12-12 11:57 /scratch/kmeans/clusters-0drwxr-xr-x - userid supergroup 0 2013-12-12 11:58 /scratch/kmeans/clusters-1-final
./mahout clusterdump -i /scratch/kmeans/clusters-*-final -d /scratch/VectorizedOutputSparce/dictionary.file-0 -dt sequencefile -b 100 -n 2013/12/12 12:28:43 INFO clustering.ClusterDumper: Wrote 7 clusters13/12/12 12:28:43 INFO driver.MahoutDriver: Program took 1066 ms (Minutes: 0.017766666666666667)
Running on hadoop, using /users/p529444/software/hadoop-1.0.3/bin/hadoop and HADOOP_CONF_DIR=/apps/hadoop/hadoop-confMAHOUT-JOB: /apps/mahout/trunk/examples/target/mahout-examples-0.9-SNAPSHOT-job.jarWarning: $HADOOP_HOME is deprecated.
13/12/16 12:52:49 INFO common.AbstractJob: Command line arguments: {--dictionary=[/scratch/VectorizedOutputSparce/dictionary.file-0], --dictionaryType=[sequencefile], --distanceMeasure=[org.apache.mahout.common.distance.SquaredEuclideanDistanceMeasure], --endPhase=[2147483647], --input=[/scratch/kmeans/clusters-*-final], --numWords=[20], --outputFormat=[TEXT], --startPhase=[0], --substring=[100], --tempDir=[temp]}:VL-0{n=2 c=[0:13.950, 1:4.932, 110:3.186, 1107280:3.186, 1118:3.186, 1177:3.186, 120:2.612, 12065996 Top Terms: records => 23.841001510620117 output => 23.411399841308594 org => 19.115327835083008 bytes => 19.115327835083008 apache => 17.00798797607422 map => 15.120783805847168 counter => 14.424721717834473 counters => 14.247724533081055 hadoop => 14.189364433288574 0 => 13.949626922607422 input => 13.516578674316406 task_type => 12.743552207946777 taskid => 12.743552207946777 file_bytes_written => 12.743552207946777 filesystemcounters => 12.743552207946777 memory => 12.743552207946777 task => 12.338891983032227 _0_b => 11.036240577697754 _1_au => 11.036240577697754 _2_vectorizedinput => 11.036240577697754:VL-1{n=2 c=[0:6.975, 0.0.0.0:6.758, 0.05:3.186, 0.11.1:6.758, 0.9:7.124, 1:6.430, 1.0:3.186, 1.0.3:1 Top Terms: name => 59.7300910949707 property => 59.68759536743164 value => 59.68759536743164 jar => 17.303800582885742 users => 17.156532287597656 software => 17.00798797607422 1.0.3 => 16.858135223388672 lib => 16.625680923461914 libexec => 16.08794593811035 hadoop => 15.67484188079834 tmp => 15.607600212097168 userid => 14.545638084411621 true => 12.338891983032227 temp => 12.131500244140625 578898841 => 11.920500755310059 false => 10.56639575958252 institution => 10.323457717895508 usr => 9.81956672668457 openjdk => 9.557663917541504 1.7.0 => 9.288379669189453
> Date: Mon, 16 Dec 2013 12:13:52 -0800
> Subject: Re: Data Vectorization
> From: andrew.musselman@gmail.com
> To: user@mahout.apache.org
>
> Looks reasonable. Does it work?
>
>
> On Mon, Dec 16, 2013 at 12:09 PM, Sameer Tilak <ss...@live.com> wrote:
>
> > Hi All,
> > I have some questions regarding vectorization.
> >
> > Here is my Pig script snippet.
> >
> > AU = FOREACH A GENERATE myparser.myUDF(param1, param2); STORE AU into
> > '/scratch/AU';
> > AU has the following format:
> > (userid, (item_view_history))
> > (27,(0,1,1,0,0))(28,(0,0,1,0,0))(29,(0,0,1,0,1))(30,(1,0,1,0,1))
> > I will have at least few hundred thousand numbers in the
> > (item_view_history), for readability I am just showing 5 here.
> > I am not sure about how to get this data written to a format that Mahout's
> > clustering algorithms will be able to parse. I have the following steps,
> > but not sure if my understanding is correct. Any help with this will be
> > great!
> >
> > VectorizedInput = FOREACH AU GENERATE FLATTEN($0);
> >
> > /*I am assuming the filed userid will be used as a key and will be written
> > using $INT_CONVERTER', and the tuple will be written using
> > $VECTOR_CONVERTER'. Is this correct?
> > STORE VectorizedInput into '/scratch/VectorizedInput' using
> > $SEQFILE_STORAGE ('-c $INT_CONVERTER', '-c $VECTOR_CONVERTER');
> >
RE: Data Vectorization
Posted by Sameer Tilak <ss...@live.com>.
Trying to figure that out now. Will keep you posted.
> Date: Mon, 16 Dec 2013 12:13:52 -0800
> Subject: Re: Data Vectorization
> From: andrew.musselman@gmail.com
> To: user@mahout.apache.org
>
> Looks reasonable. Does it work?
>
>
> On Mon, Dec 16, 2013 at 12:09 PM, Sameer Tilak <ss...@live.com> wrote:
>
> > Hi All,
> > I have some questions regarding vectorization.
> >
> > Here is my Pig script snippet.
> >
> > AU = FOREACH A GENERATE myparser.myUDF(param1, param2); STORE AU into
> > '/scratch/AU';
> > AU has the following format:
> > (userid, (item_view_history))
> > (27,(0,1,1,0,0))(28,(0,0,1,0,0))(29,(0,0,1,0,1))(30,(1,0,1,0,1))
> > I will have at least few hundred thousand numbers in the
> > (item_view_history), for readability I am just showing 5 here.
> > I am not sure about how to get this data written to a format that Mahout's
> > clustering algorithms will be able to parse. I have the following steps,
> > but not sure if my understanding is correct. Any help with this will be
> > great!
> >
> > VectorizedInput = FOREACH AU GENERATE FLATTEN($0);
> >
> > /*I am assuming the filed userid will be used as a key and will be written
> > using $INT_CONVERTER', and the tuple will be written using
> > $VECTOR_CONVERTER'. Is this correct?
> > STORE VectorizedInput into '/scratch/VectorizedInput' using
> > $SEQFILE_STORAGE ('-c $INT_CONVERTER', '-c $VECTOR_CONVERTER');
> >
Re: Data Vectorization
Posted by Andrew Musselman <an...@gmail.com>.
Looks reasonable. Does it work?
On Mon, Dec 16, 2013 at 12:09 PM, Sameer Tilak <ss...@live.com> wrote:
> Hi All,
> I have some questions regarding vectorization.
>
> Here is my Pig script snippet.
>
> AU = FOREACH A GENERATE myparser.myUDF(param1, param2); STORE AU into
> '/scratch/AU';
> AU has the following format:
> (userid, (item_view_history))
> (27,(0,1,1,0,0))(28,(0,0,1,0,0))(29,(0,0,1,0,1))(30,(1,0,1,0,1))
> I will have at least few hundred thousand numbers in the
> (item_view_history), for readability I am just showing 5 here.
> I am not sure about how to get this data written to a format that Mahout's
> clustering algorithms will be able to parse. I have the following steps,
> but not sure if my understanding is correct. Any help with this will be
> great!
>
> VectorizedInput = FOREACH AU GENERATE FLATTEN($0);
>
> /*I am assuming the filed userid will be used as a key and will be written
> using $INT_CONVERTER', and the tuple will be written using
> $VECTOR_CONVERTER'. Is this correct?
> STORE VectorizedInput into '/scratch/VectorizedInput' using
> $SEQFILE_STORAGE ('-c $INT_CONVERTER', '-c $VECTOR_CONVERTER');
>