You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by beneo_7 <be...@163.com> on 2011/10/12 12:21:05 UTC

where is the key in sequenceFile use seq2sparse

hi all
    i create vector using lucene index, and the mahout will use NamedVector, but how about create vector from sequenceFile???

    now, i create vector from text with the follow steps:

    step #1
        text -> sequeneceFile
            key = text, value = text
            i do not use seqdirectory, cuz i want to put the String key into the sequenceFile, not the doc Id

    step #2
        seq2sparse using TFIDF
            the output i use tfidf-vectors/

    step #3 #4
        canopy -> kmeans

    step #4
        clusterDump
       
        i found the vector is org.apache.mahout.math.RandomAccessSparseVector, and where i can found the sequenceFile key??

    thx in advance

Re:Re: where is the key in sequenceFile use seq2sparse

Posted by beneo_7 <be...@163.com>.
thx

At 2011-10-12 21:51:51,"Grant Ingersoll" <gs...@apache.org> wrote:
>
>On Oct 12, 2011, at 9:26 AM, beneo_7 wrote:
>
>> 
>> 
>> // sequenceFile -> vector
>> mahout seq2sparse -i ../temp/input -o ../temp/vector/ -chunk 100 -wt TFIDF -ow
>
>I think you need the --namedVector option to get/keep named vectors.  You might try using the SequenceFile dumper (seqdumper) to examine the output of this.
>
>(Also, in the future, this question is best asked on user@mahout.apache.org)
>
>> 
>> 
>> // vector -> canopy
>> mahoutcanopy -i /home/hduser/temp/vector/vector -o /home/hduser/temp/canopy/ -dm org.apache.mahout.common.distance.CosineDistanceMeasure -t1 0.032 -t2 0.008 -ow 
>> 
>> 
>> 
>> 
>> // canopy -> kmeans
>> KMeansDriver.run( conf, // configuration vectorPath, // the directory pathname for input points canopyClusterPath, // the directory pathname for initial & computed clusters kmeansPath, // the directory pathname for output points new CosineDistanceMeasure(), // cos 0.1d, // the convergence delta value 10, // the maximum number of iterations true, // run clustering false // execute map reduce );
>> 
>> 
>> 
>> 
>> no exception  thrown and thx in advance
>> 
>> 
>> 
>> 
>> At 2011-10-12 20:27:19,"Grant Ingersoll" <gs...@apache.org> wrote:
>>> Can you share your actual commands?
>>> 
>>> On Oct 12, 2011, at 6:21 AM, beneo_7 wrote:
>>> 
>>>> hi all
>>>>   i create vector using lucene index, and the mahout will use NamedVector, but how about create vector from sequenceFile???
>>>> 
>>>>   now, i create vector from text with the follow steps:
>>>> 
>>>>   step #1
>>>>       text -> sequeneceFile
>>>>           key = text, value = text
>>>>           i do not use seqdirectory, cuz i want to put the String key into the sequenceFile, not the doc Id
>>>> 
>>>>   step #2
>>>>       seq2sparse using TFIDF
>>>>           the output i use tfidf-vectors/
>>>> 
>>>>   step #3 #4
>>>>       canopy -> kmeans
>>>> 
>>>>   step #4
>>>>       clusterDump
>>>> 
>>>>       i found the vector is org.apache.mahout.math.RandomAccessSparseVector, and where i can found the sequenceFile key??
>>>> 
>>>>   thx in advance
>>> 
>>> --------------------------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com
>>> Lucene Eurocon 2011: http://www.lucene-eurocon.com
>>> 
>
>--------------------------------------------
>Grant Ingersoll
>http://www.lucidimagination.com
>Lucene Eurocon 2011: http://www.lucene-eurocon.com
>

Re: where is the key in sequenceFile use seq2sparse

Posted by Grant Ingersoll <gs...@apache.org>.
On Oct 12, 2011, at 9:26 AM, beneo_7 wrote:

> 
> 
> // sequenceFile -> vector
> mahout seq2sparse -i ../temp/input -o ../temp/vector/ -chunk 100 -wt TFIDF -ow

I think you need the --namedVector option to get/keep named vectors.  You might try using the SequenceFile dumper (seqdumper) to examine the output of this.

(Also, in the future, this question is best asked on user@mahout.apache.org)

> 
> 
> // vector -> canopy
> mahoutcanopy -i /home/hduser/temp/vector/vector -o /home/hduser/temp/canopy/ -dm org.apache.mahout.common.distance.CosineDistanceMeasure -t1 0.032 -t2 0.008 -ow 
> 
> 
> 
> 
> // canopy -> kmeans
> KMeansDriver.run( conf, // configuration vectorPath, // the directory pathname for input points canopyClusterPath, // the directory pathname for initial & computed clusters kmeansPath, // the directory pathname for output points new CosineDistanceMeasure(), // cos 0.1d, // the convergence delta value 10, // the maximum number of iterations true, // run clustering false // execute map reduce );
> 
> 
> 
> 
> no exception  thrown and thx in advance
> 
> 
> 
> 
> At 2011-10-12 20:27:19,"Grant Ingersoll" <gs...@apache.org> wrote:
>> Can you share your actual commands?
>> 
>> On Oct 12, 2011, at 6:21 AM, beneo_7 wrote:
>> 
>>> hi all
>>>   i create vector using lucene index, and the mahout will use NamedVector, but how about create vector from sequenceFile???
>>> 
>>>   now, i create vector from text with the follow steps:
>>> 
>>>   step #1
>>>       text -> sequeneceFile
>>>           key = text, value = text
>>>           i do not use seqdirectory, cuz i want to put the String key into the sequenceFile, not the doc Id
>>> 
>>>   step #2
>>>       seq2sparse using TFIDF
>>>           the output i use tfidf-vectors/
>>> 
>>>   step #3 #4
>>>       canopy -> kmeans
>>> 
>>>   step #4
>>>       clusterDump
>>> 
>>>       i found the vector is org.apache.mahout.math.RandomAccessSparseVector, and where i can found the sequenceFile key??
>>> 
>>>   thx in advance
>> 
>> --------------------------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>> Lucene Eurocon 2011: http://www.lucene-eurocon.com
>> 

--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Lucene Eurocon 2011: http://www.lucene-eurocon.com


Re:Re: where is the key in sequenceFile use seq2sparse

Posted by beneo_7 <be...@163.com>.

// sequenceFile -> vector
mahout seq2sparse -i ../temp/input -o ../temp/vector/ -chunk 100 -wt TFIDF -ow


// vector -> canopy
mahoutcanopy -i /home/hduser/temp/vector/vector -o /home/hduser/temp/canopy/ -dm org.apache.mahout.common.distance.CosineDistanceMeasure -t1 0.032 -t2 0.008 -ow 




// canopy -> kmeans
KMeansDriver.run( conf, // configuration vectorPath, // the directory pathname for input points canopyClusterPath, // the directory pathname for initial & computed clusters kmeansPath, // the directory pathname for output points new CosineDistanceMeasure(), // cos 0.1d, // the convergence delta value 10, // the maximum number of iterations true, // run clustering false // execute map reduce );




no exception  thrown and thx in advance




At 2011-10-12 20:27:19,"Grant Ingersoll" <gs...@apache.org> wrote:
>Can you share your actual commands?
>
>On Oct 12, 2011, at 6:21 AM, beneo_7 wrote:
>
>> hi all
>>    i create vector using lucene index, and the mahout will use NamedVector, but how about create vector from sequenceFile???
>> 
>>    now, i create vector from text with the follow steps:
>> 
>>    step #1
>>        text -> sequeneceFile
>>            key = text, value = text
>>            i do not use seqdirectory, cuz i want to put the String key into the sequenceFile, not the doc Id
>> 
>>    step #2
>>        seq2sparse using TFIDF
>>            the output i use tfidf-vectors/
>> 
>>    step #3 #4
>>        canopy -> kmeans
>> 
>>    step #4
>>        clusterDump
>> 
>>        i found the vector is org.apache.mahout.math.RandomAccessSparseVector, and where i can found the sequenceFile key??
>> 
>>    thx in advance
>
>--------------------------------------------
>Grant Ingersoll
>http://www.lucidimagination.com
>Lucene Eurocon 2011: http://www.lucene-eurocon.com
>

Re: where is the key in sequenceFile use seq2sparse

Posted by Grant Ingersoll <gs...@apache.org>.
Can you share your actual commands?

On Oct 12, 2011, at 6:21 AM, beneo_7 wrote:

> hi all
>    i create vector using lucene index, and the mahout will use NamedVector, but how about create vector from sequenceFile???
> 
>    now, i create vector from text with the follow steps:
> 
>    step #1
>        text -> sequeneceFile
>            key = text, value = text
>            i do not use seqdirectory, cuz i want to put the String key into the sequenceFile, not the doc Id
> 
>    step #2
>        seq2sparse using TFIDF
>            the output i use tfidf-vectors/
> 
>    step #3 #4
>        canopy -> kmeans
> 
>    step #4
>        clusterDump
> 
>        i found the vector is org.apache.mahout.math.RandomAccessSparseVector, and where i can found the sequenceFile key??
> 
>    thx in advance

--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Lucene Eurocon 2011: http://www.lucene-eurocon.com