You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Oleksandr Petrov (JIRA)" <ji...@apache.org> on 2010/10/04 09:49:33 UTC
[jira] Created: (MAHOUT-522) Using different data sources for
input/output
Using different data sources for input/output
---------------------------------------------
Key: MAHOUT-522
URL: https://issues.apache.org/jira/browse/MAHOUT-522
Project: Mahout
Issue Type: Improvement
Components: Utils
Reporter: Oleksandr Petrov
Hi,
Mahout is currently bound to the file system, at least from my feeling. Most of the time data structures i'm working with aren't located on the file system, the same way as output isn't bound to the FS, most of time i'm forced to export my datasets from DB to FS, and then load them back to DB afterwards.
Most likely, it's not quite interesting for the core developers, who're working on the algorithms implementation to start writing adapters to DBs or anything like that.
For instance, SequenceFilesFromDirectory is a simple way to get your files from directory and convert it all to Sequence Files. Some people would be extremely grateful if there would be an interface they may implement to throw their files from DB straight to the Sequence File without a medium of a File System. If anyone's interested, i can provide a patch.
Second issue is related to the workflow process itself. For instance, what if i already do have Dictionary, TF-IDF, and TF in some particular format that was created by other things in my infrastructure. Again, I need to convert those to the Mahout data-structures. Can't we just allow other jobs to accept more generic types (or interfaces, for instance) when working with TF-IDF, TF and Dictionaries, without binding those to Hadoop FS.
I do realize that Mahout is a part of Lucene/Hadoop infrastructure, but it's also an independent project, so it may benefit and get a more wide adoption, if it allows to work with any format. I do have an idea of how to implement it, and partially implemented it for our infrastructure needs, but i really want to hear some output from users and hadoop developers, whether it's suitable and if anyone may benefit out of that.
Thank you!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-522) Using different data sources for
input/output
Posted by "Oleksandr Petrov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920304#action_12920304 ]
Oleksandr Petrov commented on MAHOUT-522:
-----------------------------------------
Thank you Ted, i did. I assume that was FeatureVectorEncoder, if not, i couldn't find it.
Sorry for delay with answer, i just wanted to present some more or less representative set of thoughts.
I've been investigating Mahout code to find overall patterns. Maybe i've found an idea i've
been looking for, but still it'd be nice to discuss it.
If we take a look on (just for instance) on SequenceFilesFrom directory. That class contains
ChunkedWriter (which is a great idea), but that chunked writer isn't used in DictionaryVectorizer
(which is another possible place of usage). The overall idea is that ChunkedWriter may be a subclass
of a more generic type (interface/abstract class) which would be simply called Writer.
SequenceFilesFromDirectory also does some reading. That may be applicable for the same exact thing.
We may create an interface, that does reading, and returns current text, just as it's implemented
right now: http://gist.github.com/620827. Creating that interfaces will allow people to write their
own custom interfaces for reading.
Readers should definately be generic, since in case with SequenceFilesFromDirectory it should retrieve
Text/Text pair, with DocumentProcessor - Text/StringTuple etc.
Same exact thing is applicable to DocumentProcessor.tokenizeDocuments. Althuogh, since here we have an
already processed text (which we got as an output of SequenceFilesFromDirectory), reader should read pairs
from the existing data source in a different manner. For instance, if File Identifier (which is text in current
case) and StringTuple array, might be stored in relational DB or any other data source for that matter.
Same thing is applicable to DictionaryVectorizer.createTermFrequencyVectors. Here we have different frequency
vectors, each of those has their own format.
Please let me know if it does make some sense, and that contribution might be useful in any way, and if
anyone might want to help with that.
Thank you in advance
> Using different data sources for input/output
> ---------------------------------------------
>
> Key: MAHOUT-522
> URL: https://issues.apache.org/jira/browse/MAHOUT-522
> Project: Mahout
> Issue Type: Improvement
> Components: Utils
> Reporter: Oleksandr Petrov
>
> Hi,
> Mahout is currently bound to the file system, at least from my feeling. Most of the time data structures i'm working with aren't located on the file system, the same way as output isn't bound to the FS, most of time i'm forced to export my datasets from DB to FS, and then load them back to DB afterwards.
> Most likely, it's not quite interesting for the core developers, who're working on the algorithms implementation to start writing adapters to DBs or anything like that.
> For instance, SequenceFilesFromDirectory is a simple way to get your files from directory and convert it all to Sequence Files. Some people would be extremely grateful if there would be an interface they may implement to throw their files from DB straight to the Sequence File without a medium of a File System. If anyone's interested, i can provide a patch.
> Second issue is related to the workflow process itself. For instance, what if i already do have Dictionary, TF-IDF, and TF in some particular format that was created by other things in my infrastructure. Again, I need to convert those to the Mahout data-structures. Can't we just allow other jobs to accept more generic types (or interfaces, for instance) when working with TF-IDF, TF and Dictionaries, without binding those to Hadoop FS.
> I do realize that Mahout is a part of Lucene/Hadoop infrastructure, but it's also an independent project, so it may benefit and get a more wide adoption, if it allows to work with any format. I do have an idea of how to implement it, and partially implemented it for our infrastructure needs, but i really want to hear some output from users and hadoop developers, whether it's suitable and if anyone may benefit out of that.
> Thank you!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-522) Using different data sources for
input/output
Posted by "Oleksandr Petrov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917514#action_12917514 ]
Oleksandr Petrov commented on MAHOUT-522:
-----------------------------------------
Ted, that was exactly my point: I realize that mahout is very Hadoop-oriernted, since it's a part of its ecosystem. That was a part of my choice of Mahout. I tried Maui, which relies on Weka and Weka 3 itself, they both are great, but Mahout turned out to be better for my needs, partly, because of built-in map/reduce features.
Ok, I'll try to make patches a bit more human-oriented, or will just provide a tutorial for people who want to do things the same way I've done them to save some of their investigation time.
Thanks for response.
> Using different data sources for input/output
> ---------------------------------------------
>
> Key: MAHOUT-522
> URL: https://issues.apache.org/jira/browse/MAHOUT-522
> Project: Mahout
> Issue Type: Improvement
> Components: Utils
> Reporter: Oleksandr Petrov
>
> Hi,
> Mahout is currently bound to the file system, at least from my feeling. Most of the time data structures i'm working with aren't located on the file system, the same way as output isn't bound to the FS, most of time i'm forced to export my datasets from DB to FS, and then load them back to DB afterwards.
> Most likely, it's not quite interesting for the core developers, who're working on the algorithms implementation to start writing adapters to DBs or anything like that.
> For instance, SequenceFilesFromDirectory is a simple way to get your files from directory and convert it all to Sequence Files. Some people would be extremely grateful if there would be an interface they may implement to throw their files from DB straight to the Sequence File without a medium of a File System. If anyone's interested, i can provide a patch.
> Second issue is related to the workflow process itself. For instance, what if i already do have Dictionary, TF-IDF, and TF in some particular format that was created by other things in my infrastructure. Again, I need to convert those to the Mahout data-structures. Can't we just allow other jobs to accept more generic types (or interfaces, for instance) when working with TF-IDF, TF and Dictionaries, without binding those to Hadoop FS.
> I do realize that Mahout is a part of Lucene/Hadoop infrastructure, but it's also an independent project, so it may benefit and get a more wide adoption, if it allows to work with any format. I do have an idea of how to implement it, and partially implemented it for our infrastructure needs, but i really want to hear some output from users and hadoop developers, whether it's suitable and if anyone may benefit out of that.
> Thank you!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-522) Using different data sources for
input/output
Posted by "Oleksandr Petrov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917608#action_12917608 ]
Oleksandr Petrov commented on MAHOUT-522:
-----------------------------------------
@Drew
API is an awesome idea.
I'll probably polish my stuff that's related to Sequence files creation and TF, TF/IDF, Dictionary import. I realize that existing file format is the best thing for map/reduce usage. But still, as you mentioned, API would speed up development process a lot.
There may be an input reader, that will provide an easy iterable interfrase (getNext, maybe - getCount) for all four of mentioned things. Plus several sample adapters: MongoDB and SQL one, just for instance.
I do agree that there's no such thing as a steady DB format for such things. Everyone uses his own schema. So, moving from easy to difficult would be:
a) allow people to use / reuse / provide their own readers
b) allow single configuration point (which is questionable, since people may want to handle it all themselves)
c) implement a single writer interface that will accept reader type and read possible items out of it
We have bits of that already covered, and it surely needs to become a bit more generic to allow reuse. I'll be working on it throughout the next week or so.
> Using different data sources for input/output
> ---------------------------------------------
>
> Key: MAHOUT-522
> URL: https://issues.apache.org/jira/browse/MAHOUT-522
> Project: Mahout
> Issue Type: Improvement
> Components: Utils
> Reporter: Oleksandr Petrov
>
> Hi,
> Mahout is currently bound to the file system, at least from my feeling. Most of the time data structures i'm working with aren't located on the file system, the same way as output isn't bound to the FS, most of time i'm forced to export my datasets from DB to FS, and then load them back to DB afterwards.
> Most likely, it's not quite interesting for the core developers, who're working on the algorithms implementation to start writing adapters to DBs or anything like that.
> For instance, SequenceFilesFromDirectory is a simple way to get your files from directory and convert it all to Sequence Files. Some people would be extremely grateful if there would be an interface they may implement to throw their files from DB straight to the Sequence File without a medium of a File System. If anyone's interested, i can provide a patch.
> Second issue is related to the workflow process itself. For instance, what if i already do have Dictionary, TF-IDF, and TF in some particular format that was created by other things in my infrastructure. Again, I need to convert those to the Mahout data-structures. Can't we just allow other jobs to accept more generic types (or interfaces, for instance) when working with TF-IDF, TF and Dictionaries, without binding those to Hadoop FS.
> I do realize that Mahout is a part of Lucene/Hadoop infrastructure, but it's also an independent project, so it may benefit and get a more wide adoption, if it allows to work with any format. I do have an idea of how to implement it, and partially implemented it for our infrastructure needs, but i really want to hear some output from users and hadoop developers, whether it's suitable and if anyone may benefit out of that.
> Thank you!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-522) Using different data sources for
input/output
Posted by "Drew Farris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917582#action_12917582 ]
Drew Farris commented on MAHOUT-522:
------------------------------------
Yes Oleksandr, please share some patches. It would be great to see your ideas in implementation, even if they are rough sketches of ideas in code.
As far as database input is concerned, it's difficult to predict just what format or schema any one potential mahout user's data is stored. To that end it makes a great deal of sense to move towards api oriented access, Mahout needs to provide api's for reading and writing various bits of data while users implement the necessary bits of glue to hook them up to their data sources.
You don't mention what portion of Mahout you're most interested in, where do you propose to get started?
> Using different data sources for input/output
> ---------------------------------------------
>
> Key: MAHOUT-522
> URL: https://issues.apache.org/jira/browse/MAHOUT-522
> Project: Mahout
> Issue Type: Improvement
> Components: Utils
> Reporter: Oleksandr Petrov
>
> Hi,
> Mahout is currently bound to the file system, at least from my feeling. Most of the time data structures i'm working with aren't located on the file system, the same way as output isn't bound to the FS, most of time i'm forced to export my datasets from DB to FS, and then load them back to DB afterwards.
> Most likely, it's not quite interesting for the core developers, who're working on the algorithms implementation to start writing adapters to DBs or anything like that.
> For instance, SequenceFilesFromDirectory is a simple way to get your files from directory and convert it all to Sequence Files. Some people would be extremely grateful if there would be an interface they may implement to throw their files from DB straight to the Sequence File without a medium of a File System. If anyone's interested, i can provide a patch.
> Second issue is related to the workflow process itself. For instance, what if i already do have Dictionary, TF-IDF, and TF in some particular format that was created by other things in my infrastructure. Again, I need to convert those to the Mahout data-structures. Can't we just allow other jobs to accept more generic types (or interfaces, for instance) when working with TF-IDF, TF and Dictionaries, without binding those to Hadoop FS.
> I do realize that Mahout is a part of Lucene/Hadoop infrastructure, but it's also an independent project, so it may benefit and get a more wide adoption, if it allows to work with any format. I do have an idea of how to implement it, and partially implemented it for our infrastructure needs, but i really want to hear some output from users and hadoop developers, whether it's suitable and if anyone may benefit out of that.
> Thank you!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-522) Using different data sources for
input/output
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917512#action_12917512 ]
Ted Dunning commented on MAHOUT-522:
------------------------------------
If you want a classifier framework that is more API oriented, take a look at the o.a.m.classifier.sgd package. These provide in-memory sequential, multi-threaded implementations of logistic regression.
Your point about the entire system being a bit limited is spot on. The Naive Bayes models do have a generic interface for saving models and reloading them, but the Hadoop based parts of Mahout are heavily dependent on file systems which is a characteristic that is easy to inherit from the assumptions of running code with Hadoop.
That said, I am sure that if you have concrete suggestions or, better yet, patches, they would be very welcome. Don't feel you have to build a final answer, just try something. IF there is a spark there, we will help start a fire.
> Using different data sources for input/output
> ---------------------------------------------
>
> Key: MAHOUT-522
> URL: https://issues.apache.org/jira/browse/MAHOUT-522
> Project: Mahout
> Issue Type: Improvement
> Components: Utils
> Reporter: Oleksandr Petrov
>
> Hi,
> Mahout is currently bound to the file system, at least from my feeling. Most of the time data structures i'm working with aren't located on the file system, the same way as output isn't bound to the FS, most of time i'm forced to export my datasets from DB to FS, and then load them back to DB afterwards.
> Most likely, it's not quite interesting for the core developers, who're working on the algorithms implementation to start writing adapters to DBs or anything like that.
> For instance, SequenceFilesFromDirectory is a simple way to get your files from directory and convert it all to Sequence Files. Some people would be extremely grateful if there would be an interface they may implement to throw their files from DB straight to the Sequence File without a medium of a File System. If anyone's interested, i can provide a patch.
> Second issue is related to the workflow process itself. For instance, what if i already do have Dictionary, TF-IDF, and TF in some particular format that was created by other things in my infrastructure. Again, I need to convert those to the Mahout data-structures. Can't we just allow other jobs to accept more generic types (or interfaces, for instance) when working with TF-IDF, TF and Dictionaries, without binding those to Hadoop FS.
> I do realize that Mahout is a part of Lucene/Hadoop infrastructure, but it's also an independent project, so it may benefit and get a more wide adoption, if it allows to work with any format. I do have an idea of how to implement it, and partially implemented it for our infrastructure needs, but i really want to hear some output from users and hadoop developers, whether it's suitable and if anyone may benefit out of that.
> Thank you!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-522) Using different data sources for
input/output
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917616#action_12917616 ]
Ted Dunning commented on MAHOUT-522:
------------------------------------
Oleksandr,
Take a look at FeatureValueEncoder style vector creation as well. It might be helpful to you.
> Using different data sources for input/output
> ---------------------------------------------
>
> Key: MAHOUT-522
> URL: https://issues.apache.org/jira/browse/MAHOUT-522
> Project: Mahout
> Issue Type: Improvement
> Components: Utils
> Reporter: Oleksandr Petrov
>
> Hi,
> Mahout is currently bound to the file system, at least from my feeling. Most of the time data structures i'm working with aren't located on the file system, the same way as output isn't bound to the FS, most of time i'm forced to export my datasets from DB to FS, and then load them back to DB afterwards.
> Most likely, it's not quite interesting for the core developers, who're working on the algorithms implementation to start writing adapters to DBs or anything like that.
> For instance, SequenceFilesFromDirectory is a simple way to get your files from directory and convert it all to Sequence Files. Some people would be extremely grateful if there would be an interface they may implement to throw their files from DB straight to the Sequence File without a medium of a File System. If anyone's interested, i can provide a patch.
> Second issue is related to the workflow process itself. For instance, what if i already do have Dictionary, TF-IDF, and TF in some particular format that was created by other things in my infrastructure. Again, I need to convert those to the Mahout data-structures. Can't we just allow other jobs to accept more generic types (or interfaces, for instance) when working with TF-IDF, TF and Dictionaries, without binding those to Hadoop FS.
> I do realize that Mahout is a part of Lucene/Hadoop infrastructure, but it's also an independent project, so it may benefit and get a more wide adoption, if it allows to work with any format. I do have an idea of how to implement it, and partially implemented it for our infrastructure needs, but i really want to hear some output from users and hadoop developers, whether it's suitable and if anyone may benefit out of that.
> Thank you!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-522) Using different data sources for
input/output
Posted by "Robin Anil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917545#action_12917545 ]
Robin Anil commented on MAHOUT-522:
-----------------------------------
+1
> Using different data sources for input/output
> ---------------------------------------------
>
> Key: MAHOUT-522
> URL: https://issues.apache.org/jira/browse/MAHOUT-522
> Project: Mahout
> Issue Type: Improvement
> Components: Utils
> Reporter: Oleksandr Petrov
>
> Hi,
> Mahout is currently bound to the file system, at least from my feeling. Most of the time data structures i'm working with aren't located on the file system, the same way as output isn't bound to the FS, most of time i'm forced to export my datasets from DB to FS, and then load them back to DB afterwards.
> Most likely, it's not quite interesting for the core developers, who're working on the algorithms implementation to start writing adapters to DBs or anything like that.
> For instance, SequenceFilesFromDirectory is a simple way to get your files from directory and convert it all to Sequence Files. Some people would be extremely grateful if there would be an interface they may implement to throw their files from DB straight to the Sequence File without a medium of a File System. If anyone's interested, i can provide a patch.
> Second issue is related to the workflow process itself. For instance, what if i already do have Dictionary, TF-IDF, and TF in some particular format that was created by other things in my infrastructure. Again, I need to convert those to the Mahout data-structures. Can't we just allow other jobs to accept more generic types (or interfaces, for instance) when working with TF-IDF, TF and Dictionaries, without binding those to Hadoop FS.
> I do realize that Mahout is a part of Lucene/Hadoop infrastructure, but it's also an independent project, so it may benefit and get a more wide adoption, if it allows to work with any format. I do have an idea of how to implement it, and partially implemented it for our infrastructure needs, but i really want to hear some output from users and hadoop developers, whether it's suitable and if anyone may benefit out of that.
> Thank you!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.