You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Thomas Jungblut (JIRA)" <ji...@apache.org> on 2012/09/11 18:35:07 UTC
[jira] [Updated] (HAMA-619) Hama Pipes
[ https://issues.apache.org/jira/browse/HAMA-619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-619:
---------------------------------
Attachment: HAMA-619.patch
Hey Martin, I just generated a patch for the java part. Would like to start with streaming soon, so I need to finalize the protocol.
I have a question about the implementation of the read and write from and to sequence files.
Why aren't you using the plain "WRITE_KEYVALUE" method?
You can configure the job to output sequence files, there is no need to implement the stuff on your own.
Personally I would delete this, but I don't know what parts of your implementations are grounding on it.
> Hama Pipes
> ----------
>
> Key: HAMA-619
> URL: https://issues.apache.org/jira/browse/HAMA-619
> Project: Hama
> Issue Type: New Feature
> Components: bsp core, messaging
> Affects Versions: 0.6.0
> Reporter: Martin Illecker
> Assignee: Martin Illecker
> Attachments: HAMA-619.patch
>
>
> *Adapting Hadoop Pipes to Hama for C++ integration.*
> The package *org.apache.hama.pipes* contains all pipes specific code. The C++ implementation is located in the *c++* folder under *pipes* and *utils*. There are also some minor modifications in other packages but you can checkout the full Hama source here: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/]
> The changelog can be found here: [here|http://hadoop.illecker.at/?page_id=6]
> Hama Pipes: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/core/src/main/java/org/apache/hama/pipes/]
> Hama Pipes C++: [https://svn.illecker.at/hadoop/hama-0.5.0-gpu/c++/pipes/]
> Hama Pipes provides the following methods for C++ integration: (similar to Hama BSP)
> *{{virtual void sendMessage(const string& peerName, const string& msg) = 0;}}*
> bq. Send a data with a tag to another BSPSlave corresponding to hostname. Messages sent by this method are not guaranteed to be received in a sent order.
> *{{virtual const string& getCurrentMessage() = 0;}}*
> bq. Returns a message from the peer's received messages queue (a FIFO).
> *{{virtual int getNumCurrentMessages() = 0;}}*
> bq. Returns the number of messages in the peer's received messages queue.
> *{{virtual void sync() = 0;}}*
> {quote}
> Barrier Synchronization.
> Sends all the messages in the outgoing message queues to the corresponding remote peers.
> {quote}
> *{{virtual long getSuperstepCount() = 0;}}*
> bq. Returns the count of current super-step
>
> *{{virtual const string& getPeerName() = 0;}}*
> bq. Returns the name of this peer in the format "hostname:port".
>
> *{{virtual const string& getPeerName(int index) = 0;}}*
> bq. Returns the name of n-th peer from sorted array by name.
>
> *{{virtual int getPeerIndex() = 0;}}*
> bq. Returns the index of this peer from sorted array by name.
> *{{virtual vector<string> getAllPeerNames() = 0;}}*
> bq. Returns the names of all the peers executing tasks from the same job (including this peer).
>
> *{{virtual int getNumPeers() = 0;}}*
> bq. Returns the number of peers
>
> *{{virtual void clear() = 0;}}*
> bq. Clears all queues entries.
>
> *{{virtual void write(const string& key, const string& value) = 0;}}*
> bq. Writes a key/value pair to the output collector
>
> *{{virtual bool readNext(string& key, string& value) = 0;}}*
> bq. Deserializes the next input key value into the given objects;
> *{{virtual void reopenInput() = 0;}}*
> bq. Closes the input and opens it right away, so that the file pointer is at the beginning again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Updated] (HAMA-619) Hama Pipes
Posted by Thomas Jungblut <th...@gmail.com>.
Ah I remember, it was because of the center writer in my kmeans example!
Sorry. However I don't think writing sequence file via a binary protocol is
smart.
Am 11.09.2012 18:35 schrieb "Thomas Jungblut (JIRA)" <ji...@apache.org>:
>
> [
> https://issues.apache.org/jira/browse/HAMA-619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Thomas Jungblut updated HAMA-619:
> ---------------------------------
>
> Attachment: HAMA-619.patch
>
> Hey Martin, I just generated a patch for the java part. Would like to
> start with streaming soon, so I need to finalize the protocol.
>
> I have a question about the implementation of the read and write from and
> to sequence files.
>
> Why aren't you using the plain "WRITE_KEYVALUE" method?
>
> You can configure the job to output sequence files, there is no need to
> implement the stuff on your own.
> Personally I would delete this, but I don't know what parts of your
> implementations are grounding on it.
>
> > Hama Pipes
> > ----------
> >
> > Key: HAMA-619
> > URL: https://issues.apache.org/jira/browse/HAMA-619
> > Project: Hama
> > Issue Type: New Feature
> > Components: bsp core, messaging
> > Affects Versions: 0.6.0
> > Reporter: Martin Illecker
> > Assignee: Martin Illecker
> > Attachments: HAMA-619.patch
> >
> >
> > *Adapting Hadoop Pipes to Hama for C++ integration.*
> > The package *org.apache.hama.pipes* contains all pipes specific code.
> The C++ implementation is located in the *c++* folder under *pipes* and
> *utils*. There are also some minor modifications in other packages but you
> can checkout the full Hama source here: [
> https://svn.illecker.at/hadoop/hama-0.5.0-gpu/]
> > The changelog can be found here: [here|
> http://hadoop.illecker.at/?page_id=6]
> > Hama Pipes: [
> https://svn.illecker.at/hadoop/hama-0.5.0-gpu/core/src/main/java/org/apache/hama/pipes/
> ]
> > Hama Pipes C++: [
> https://svn.illecker.at/hadoop/hama-0.5.0-gpu/c++/pipes/]
> > Hama Pipes provides the following methods for C++ integration: (similar
> to Hama BSP)
> > *{{virtual void sendMessage(const string& peerName, const string& msg) =
> 0;}}*
> > bq. Send a data with a tag to another BSPSlave corresponding to
> hostname. Messages sent by this method are not guaranteed to be received in
> a sent order.
> > *{{virtual const string& getCurrentMessage() = 0;}}*
> > bq. Returns a message from the peer's received messages queue (a FIFO).
> > *{{virtual int getNumCurrentMessages() = 0;}}*
> > bq. Returns the number of messages in the peer's received messages queue.
> > *{{virtual void sync() = 0;}}*
> > {quote}
> > Barrier Synchronization.
> > Sends all the messages in the outgoing message queues to the
> corresponding remote peers.
> > {quote}
> > *{{virtual long getSuperstepCount() = 0;}}*
> > bq. Returns the count of current super-step
> >
> > *{{virtual const string& getPeerName() = 0;}}*
> > bq. Returns the name of this peer in the format "hostname:port".
> >
> > *{{virtual const string& getPeerName(int index) = 0;}}*
> > bq. Returns the name of n-th peer from sorted array by name.
> >
> > *{{virtual int getPeerIndex() = 0;}}*
> > bq. Returns the index of this peer from sorted array by name.
> > *{{virtual vector<string> getAllPeerNames() = 0;}}*
> > bq. Returns the names of all the peers executing tasks from the same job
> (including this peer).
> >
> > *{{virtual int getNumPeers() = 0;}}*
> > bq. Returns the number of peers
> >
> > *{{virtual void clear() = 0;}}*
> > bq. Clears all queues entries.
> >
> > *{{virtual void write(const string& key, const string& value) = 0;}}*
> > bq. Writes a key/value pair to the output collector
> >
> > *{{virtual bool readNext(string& key, string& value) = 0;}}*
> > bq. Deserializes the next input key value into the given objects;
> > *{{virtual void reopenInput() = 0;}}*
> > bq. Closes the input and opens it right away, so that the file pointer
> is at the beginning again.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>