You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Robin Anil <ro...@gmail.com> on 2009/03/30 08:10:07 UTC
Participating in GSOC 2009
Hi, I am planning to participate again in GSOC this year and I want to
do it again under Mahout. I am finishing up my Masters in CSE from IIT
Kharagpur and will be joining Yahoo! this fall, where I would be seeing a
lot of Hadoop :). I would like to thank all of you for all the help you have
given. This year I want to primarily focus on completing the classification
module for Mahout and to . Till now we have a M/R version of NB Model
building. I want to add the following features
1) Batch classification of flat file documents and flat file model
2) Storing the model using HBase
3) Quick classification of a document using HBase Model.
4) Batch classification using HBase Model
5*) Another option I have for quick model access is to load the key-value
pairs over Memcached servers(I use this currently in my Masters thesis
work). I would like to know if this is something that is aligned with
Mahout's objectives
Apart from this I am looking at the other patches and see where they stand
and would like to attack another classification algorithm and implement the
above features for the same.
Warm Regards
Robin
Re: Participating in GSOC 2009
Posted by Ted Dunning <te...@gmail.com>.
Good luck with your party! (i.e. I hope that you have something to
celebrate)
On Sun, Apr 5, 2009 at 10:27 AM, Robin Anil <ro...@gmail.com> wrote:
> This was one of the reason I didnt apply sooner, as the actual dates of
> defense was not clear initially. So my schedule this summer is
> 20 April Thesis Submission
> 6th May Thesis Evaluation by panel
> 7th May Party :D
> 23rd May - GSOC
>
>
--
Ted Dunning, CTO
DeepDyve
Re: Participating in GSOC 2009
Posted by Robin Anil <ro...@gmail.com>.
>
> Hi Isabel
>
> Hmm, I wonder whether this might interfere with GSoC - usually writing a
> masters thesis takes up quite a bit of time. What are your exact plans for
> this issue?
>
This was one of the reason I didnt apply sooner, as the actual dates of
defense was not clear initially. So my schedule this summer is
20 April Thesis Submission
6th May Thesis Evaluation by panel
7th May Party :D
23rd May - GSOC
Robin
Re: Participating in GSOC 2009
Posted by Isabel Drost <is...@apache.org>.
Hello Robin,
On Monday 30 March 2009, Robin Anil wrote:
> Hi, I am planning to participate again in GSOC this year and I want to
> do it again under Mahout.
Nice to hear that :)
> I am finishing up my Masters in CSE from IIT Kharagpur
Hmm, I wonder whether this might interfere with GSoC - usually writing a
masters thesis takes up quite a bit of time. What are your exact plans for
this issue?
> and will be joining Yahoo! this fall, where I would be seeing a
Congratulations! Wish you a lot of fun there.
Isabel
--
A full belly makes a dull brain. -- Ben Franklin [and the local candy
machine man. Ed]
|\ _,,,---,,_ Web: <http://www.isabel-drost.de>
/,`.-'`' -. ;-;;,_
|,4- ) )-,_..;\ ( `'-'
'---''(_/--' `-'\_) (fL) IM: <xm...@spaceboyz.net>
Re: Participating in GSOC 2009
Posted by Robin Anil <ro...@gmail.com>.
I submitted my proposal.
Here it is
Detailed Description:
My proposal has two broad objectives.
- Create an an online + batch classification system for the
current NB/CNB Classifier using HBase
- Implement Parallel FP Growth algorithm and create a tag-tag
relationship from the wikipedia dataset using the same.
Deliverables:
1. Batch classification of flat file documents and flat file
model: Be able to batch classify documents and models stored on the HDFS
2. Storing the model using HBase: Storing the model in HBase and
the end of Model Building Map/Reduce stages
3. Quick classification of a document using HBase Model: Using the
model stored in HBase create an interface (both command line and web
service) to classify a give document
4. Batch classification using HBase Model: Using the model stored
in HBase, batch classify documents stored on the HDFS
5. Implement Frequent Pattern Mining using Parallel FP-Growth
Algorithm: Frequent patterns occurring in data is of interest in a wide
variety of fields. The paper titled “PFP: Parallel FP-Growth for Query
Recommendation” showed that frequent patterns can be used to generate
co-occurring search queries to which can be used as query suggestions for
users.
http://www.haoyuanli.com/publication/recsys08-69.pdf
6. Documentations and Unit tests
7. Getting started tutorials for NB, CNB and PFPGrowth
8. Out of the box demos for Naïve Bayes in model building and
classification using 20Newsgroups and Wikipedia data and for PFPGrowth with
frequently occurring word groups from wikipedia
Frequent Pattern Mining:
Frequent patterns occurring in data is of interest in a wide variety of
fields. The paper titled “PFP: Parallel FP-Growth for Query Recommendation”
showed that frequent patterns can be used to generate co-occurring search
queries to which can be used as query suggestions for users
Application of Frequent Pattern Mining is not limited to the above
mentioned.
1) One important application of frequent pattern mining is in sentiment
analysis. Suppose you want to identify a niche demographics’ likeness over a
certain object or an event, frequent pattern can bring out. Lets take an
example, suppose a survey is given on Facebook which asks users to agree or
disagree on the opinion “Pepsi is better than coke”
http://minekey.com/theopinion/1001/pepsi-is-better-than-coke. From the user
responses, we will be able to mine the frequently occurring user attributes.
I.e. we can say 90% of women from Ireland who are married disagreed with the
statement “Pepsi is better than coke”. Such a system is useful for market
researchers. Here we are doing frequent pattern mining over all users who
have voted on this opinion. I have developed such a framework at Minekey,
where I have worked on generating interesting insights from user opinion and
voting data(See below for other measures of interestingness). Though we
have not used Hadoop yet at minekey, I am pretty excited to implement
PFPGrowth which will be beneficial to the community at large.
2) Another application is analyzing user behavior from weblog data.
PFPGrowth is used as the base algorithm to generate the frequent surfing
behavior in any website. http://bmf.hu/journal/Ivancsy_Vajk_5.pdf
- Home->news->updates – 2%
- Home->shop->checkout – 1%
Other measures of interestingness
Other than traditional support/confidence measurement, there are other
measures of interestingness. If time permits, I would modify PFPGrowth to
run on the following measures.
- All-confidence
- Collective strength
- Conviction
- Coverage
- Leverage
- Lift (originally called interest)
http://michael.hahsler.net/research/association_rules/measures.html
Benefits to Community:
1) Complete Classification module, online and offline. Next release of
Mahout will go with a complete end to end classifier with both distributed
model building as well as distributed batch classification together with
online classification using Hadoop HBase.
2) Frequent Pattern Mining is an important tool for analytics and
insights generation. Map/Reduce version is really useful and will see wide
adoption in various companies.
3) Better tutorials and ease of use demos to improve Mahout adoption.
Roadmap:
1. Community Bonding Period (20th April to 23rd May)
1. Familiarize with HBase configuration.
2. Setup a test bed comprising of 2-3 system cluster.
3. Experiment with HBase row vector using HTables
MAHOUT-78<https://issues.apache.org/jira/browse/MAHOUT-78>
2. Start with HBase (23rd May to 23th June)
1. Implement File based Batch classification over files over HDFS(1
week)
2. HBase based storage for NB/CNB Model(0.5 week)
3. New Unit Tests for HBase model storage and for NB/CNB model
builder(0.5 week)
4. HBase based batch Classification (1 week).
5. HBase based online classification and Web Service classification
demo for Wikipedia(1 week)
3. 3. PFPGrowth (23th June to 6th July)
1. Implement PFPGrowth algorithm and verify correctness of
implementation (2 weeks)
4. Mid Term Evaluation:
5. PFPGrowth – Testing phase (6th July to 10th August)
1. Unit Tests for PFPGrowth(0.5 week)
2. Wikipedia demo for PFPGrowth – analyze and release initial results
for frequently co-occurring terms in wikipedia(1.5 weeks)
3. Finish up the demo. (1 week)
4. Write tutorials and documentation for NB/CNB and PFPGrowth.(rest
of the time)
5. Add other interestingness measures to PFPGrowth and check results
6. Pencil Down (10th August)
1. Code Cleanup. Finish off everything
Re: Participating in GSOC 2009
Posted by Grant Ingersoll <gs...@apache.org>.
Those added features all sounds good, Robin, would like to see a
proposal from you. I do know Allen Day submitted some HBase work in
JIRA, but I believe it needs updating. You might want to check that
out.
And good luck at Y! Hopefully your Mahout experience was helpful in
landing the job!
On Mar 30, 2009, at 2:10 AM, Robin Anil wrote:
> Hi, I am planning to participate again in GSOC this year and I
> want to
> do it again under Mahout. I am finishing up my Masters in CSE from IIT
> Kharagpur and will be joining Yahoo! this fall, where I would be
> seeing a
> lot of Hadoop :). I would like to thank all of you for all the help
> you have
> given. This year I want to primarily focus on completing the
> classification
> module for Mahout and to . Till now we have a M/R version of NB Model
> building. I want to add the following features
>
> 1) Batch classification of flat file documents and flat file model
> 2) Storing the model using HBase
> 3) Quick classification of a document using HBase Model.
> 4) Batch classification using HBase Model
>
> 5*) Another option I have for quick model access is to load the key-
> value
> pairs over Memcached servers(I use this currently in my Masters thesis
> work). I would like to know if this is something that is aligned with
> Mahout's objectives
>
> Apart from this I am looking at the other patches and see where they
> stand
> and would like to attack another classification algorithm and
> implement the
> above features for the same.
>
> Warm Regards
> Robin