You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jeremy Hanna (JIRA)" <ji...@apache.org> on 2010/09/13 17:59:32 UTC
[jira] Created: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Add input support for Hadoop Streaming
--------------------------------------
Key: CASSANDRA-1497
URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
Project: Cassandra
Issue Type: New Feature
Components: Hadoop
Reporter: Jeremy Hanna
Fix For: 0.7.0
related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1497) Add input support for
Hadoop Streaming
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923292#action_12923292 ]
Stu Hood edited comment on CASSANDRA-1497 at 10/20/10 11:00 PM:
----------------------------------------------------------------
contrib/hadoop_streaming_input/bin/mapper.py
* Mentions the original source multiple times, and claims to be both a mapper and reducer
* I suspect that extract_text can be turned into a one-liner somehow
contrib/hadoop_streaming_input/bin/reducer.py
* Needs an Apache header
contrib/hadoop_streaming_input/[input/]README.txt
* Mentions "-input": {{bin/streaming}} should fake the input, and explain why
* There is an extra copy of README.txt in an unused 'input' subdirectory
.../hadoop/ColumnFamilyRecordReader.java
* Indentation
.../hadoop/streaming/AvroResolver.java
* Updated javadoc
I looked a little bit into the immediate runtime failure, but didn't come to any conclusions. One suspicious aspect is that Streaming appears to use the result of Resolver.getInputWriterClass to write to both the mapper and reducer scripts: see http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java?view=markup#l783
was (Author: stuhood):
contrib/hadoop_streaming_input/bin/mapper.py
* Mentions the original source multiple times, and claims to be both a mapper and reducer
* I suspect that extract_text can be turned into a one-liner somehow
contrib/hadoop_streaming_input/bin/reducer.py
* Needs an Apache header
contrib/hadoop_streaming_input/[input/]README.txt
* Mentions "-input": {{bin/streaming}} should fake the input, and explain why
* There is an extra copy of README.txt in an unused 'input' subdirectory
.../hadoop/ColumnFamilyRecordReader.java
* Indentation
.../hadoop/streaming/AvroResolver.java
* Updated javadoc
I looked a little bit into the immediate runtime failure, but didn't come to any conclusions. One suspicious aspect is that Streaming appears to use the result of Resolver.getInputWriterClass to write to both the mapper and reducer scripts: see http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java?view=markup#l783
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.1
>
> Attachments: 0001-An-updated-avro-based-input-streaming-solution.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919305#action_12919305 ]
Jeremy Hanna commented on CASSANDRA-1497:
-----------------------------------------
In addition to the previous comment, I've also left the original ColumnFamilyRecordReader and ColumnFamilyInputFormat undeprecated. So there will be 3 extensions of the abstract - CFRR/CFIF, TCFRR/TCFIF, and ACFRR/ACFIF. The only code that is in the extension is in the extension classes has to do with data type marshalling, so it's not duplicated code.
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.0
>
> Attachments: 0001-1497-foundation-changes.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922327#action_12922327 ]
Jeremy Hanna commented on CASSANDRA-1497:
-----------------------------------------
stu - btw, the change in your contrib/hadoop_streaming_output/bin/streaming in the patch is wrong. that was for when I had a AvroColumnFamilyRecordReader. also you just need to make sure hadoop's bin (0.20.2+320) is in your PATH.
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.1
>
> Attachments: 0001-An-updated-avro-based-input-streaming-solution.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Hanna updated CASSANDRA-1497:
------------------------------------
Attachment: (was: 0001-foundation-for-1497.patch)
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.0
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917998#action_12917998 ]
Jeremy Hanna commented on CASSANDRA-1497:
-----------------------------------------
No - I'm still getting the o.a.c.hadoop.streaming.AvroInputWriter done. That may change this patch slightly as it turns out. The foundation patch won't change too much though.
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.0
>
> Attachments: 0001-1497-foundation-changes.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923292#action_12923292 ]
Stu Hood commented on CASSANDRA-1497:
-------------------------------------
contrib/hadoop_streaming_input/bin/mapper.py
* Mentions the original source multiple times, and claims to be both a mapper and reducer
* I suspect that extract_text can be turned into a one-liner somehow
contrib/hadoop_streaming_input/bin/reducer.py
* Needs an Apache header
contrib/hadoop_streaming_input/[input/]README.txt
* Mentions "-input": {{bin/streaming}} should fake the input, and explain why
* There is an extra copy of README.txt in an unused 'input' subdirectory
.../hadoop/ColumnFamilyRecordReader.java
* Indentation
.../hadoop/streaming/AvroResolver.java
* Updated javadoc
I looked a little bit into the immediate runtime failure, but didn't come to any conclusions. One suspicious aspect is that Streaming appears to use the result of Resolver.getInputWriterClass to write to both the mapper and reducer scripts: see http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java?view=markup#l783
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.1
>
> Attachments: 0001-An-updated-avro-based-input-streaming-solution.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Hanna updated CASSANDRA-1497:
------------------------------------
Attachment: 0001-foundation-for-1497.patch
Attaching 0001:
Deprecates the old way of doing ColumnFamilyRecordReader and ColumnFamilyInputFormat - renames them to LegacyX. Reused code as I could (in the InputFormat).
Adds the new ColumnFamilyRecordReader and ColumnFamilyInputReader that uses ByteBuffers and Avro ColumnOrSuperColumns instead of byte[] and db.IColumn. This makes for easier Hadoop Streaming support and mirrors what was done in the RecordWriter/OutputFormat that we recently added.
Updated the WordCount/WordCountSetup to use the newer code.
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.0
>
> Attachments: 0001-foundation-for-1497.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920296#action_12920296 ]
Jeremy Hanna commented on CASSANDRA-1497:
-----------------------------------------
based on a conversation with jonathan - it sounds like he doesn't want several different implementations nor does he want a mixture of avro for output and thrift for input.
For the long term I've logged this ticket with avro - https://issues.apache.org/jira/browse/AVRO-680
For the short term, we could probably get away with avro - having a list of columns (k/v pairs) that the client could scan through or convert to a map on their own.
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.1
>
> Attachments: 0001-1497-foundation-changes.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Hanna updated CASSANDRA-1497:
------------------------------------
Attachment: (was: 0001-foundation-for-1497.patch)
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.0
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917943#action_12917943 ]
Jonathan Ellis commented on CASSANDRA-1497:
-------------------------------------------
is this ready for review?
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.0
>
> Attachments: 0001-1497-foundation-changes.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Hanna updated CASSANDRA-1497:
------------------------------------
Attachment: 0001-An-updated-avro-based-input-streaming-solution.patch
Updated to use avro. Despite there being no non-string based map in avro, I just used a key/value pair list for the input values. So using a binary search (since the column names are ordered) is only O(log n) to get each column out of the results. If they use a slice predicate too, it shouldn't be bad at all.
The current patch is mostly done - some errors with the streaming script. Also docs need updating and python needs to use bisect or something similar to do the binary search over the list of columns. Currently it just iterates and searches.
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.1
>
> Attachments: 0001-An-updated-avro-based-input-streaming-solution.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916630#action_12916630 ]
Stu Hood commented on CASSANDRA-1497:
-------------------------------------
* CFIF should be able to delegate almost everything to a private LegacyCFIF: no point in duplicating things
* Other parts of the code use com.google.common.base.Charsets.UTF_8 as our reference to the UTF8 Charset
* Pig contrib should use LegacyCFIF until we can get around to changing it
Thanks!
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.0
>
> Attachments: 0001-foundation-for-1497.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1497) Add input support for
Hadoop Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919305#action_12919305 ]
Jeremy Hanna edited comment on CASSANDRA-1497 at 10/8/10 11:25 AM:
-------------------------------------------------------------------
In addition to the previous comment, I've also left the original ColumnFamilyRecordReader and ColumnFamilyInputFormat undeprecated. So there will be 3 extensions of the abstract - CFRR/CFIF, TCFRR/TCFIF, and ACFRR/ACFIF. The only code that is in the extension classes has to do with data type marshalling, so it's not duplicated code.
was (Author: jeromatron):
In addition to the previous comment, I've also left the original ColumnFamilyRecordReader and ColumnFamilyInputFormat undeprecated. So there will be 3 extensions of the abstract - CFRR/CFIF, TCFRR/TCFIF, and ACFRR/ACFIF. The only code that is in the extension is in the extension classes has to do with data type marshalling, so it's not duplicated code.
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.0
>
> Attachments: 0001-1497-foundation-changes.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Hanna updated CASSANDRA-1497:
------------------------------------
Attachment: 0001-foundation-for-1497.patch
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.0
>
> Attachments: 0001-foundation-for-1497.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Hanna updated CASSANDRA-1497:
------------------------------------
Attachment: (was: 0001-1497-foundation-changes.patch)
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.1
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Hanna updated CASSANDRA-1497:
------------------------------------
Attachment: 0001-1497-foundation-changes.patch
For the foundation, added an AbstractColumnFamilyRecordReader with most of the functionality. Deprecated the way we have been doing CFRR to LegacyCFRR and CFIF to LegacyCFIF. The new way paves the way for enabling streaming.
I had thought to move the current CFRR/CFIF to LegacyX so that people using the forward-looking stuff wouldn't have to change it once the deprecated stuff goes away. Those upgrading to 0.7 would have to modify their class they're using though.
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.0
>
> Attachments: 0001-1497-foundation-changes.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1497) Add input support for
Hadoop Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917998#action_12917998 ]
Jeremy Hanna edited comment on CASSANDRA-1497 at 10/5/10 10:40 AM:
-------------------------------------------------------------------
No - I'm still getting the o.a.c.hadoop.streaming.AvroInputWriter done. That may change this patch slightly as it turns out. The foundation patch won't change too much though. I'm hoping to get the complete patch finished today though.
was (Author: jeromatron):
No - I'm still getting the o.a.c.hadoop.streaming.AvroInputWriter done. That may change this patch slightly as it turns out. The foundation patch won't change too much though.
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.0
>
> Attachments: 0001-1497-foundation-changes.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919302#action_12919302 ]
Jeremy Hanna commented on CASSANDRA-1497:
-----------------------------------------
Just as a status update. I ran into some issues with using Avro as an input format for now - avro maps currently require strings as their keys. There is talk about facilitating other types of keys in the future - http://search-hadoop.com/m/b2Wqk1HvNbT1/+%2522type+array%2522&subj=Re+Using+bytes+as+keys+in+a+map+equivalent - and I should probably create a ticket on that.
However for now, it looks like I'll need to go with thrift for the input streaming solution. So far I've created an AbstractColumnFamilyRecordReader and AbstractColumnFamilyInputFormat that pulls out a lot of the code, and then created thrift and avro specific extensions of each. I've created a thrift input writer, required for input streaming. I've also created an initial mapper.py that uses the thrift deserialization.
Anyway, so that's where things stand. I'm getting the kinks out and will hopefully have patches soon. I'll probably create a separate ticket for a thrift based output format for sake of completeness. The avro specific CFRR and CFIF will just be there and functional for when the avro stuff becomes available - already had it done and it works in Java.
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.0
>
> Attachments: 0001-1497-foundation-changes.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1497:
--------------------------------------
Fix Version/s: (was: 0.7.0)
0.7.1
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.1
>
> Attachments: 0001-1497-foundation-changes.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Hanna reassigned CASSANDRA-1497:
---------------------------------------
Assignee: Jeremy Hanna
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.0
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1497) Add input support for Hadoop
Streaming
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922272#action_12922272 ]
Stu Hood commented on CASSANDRA-1497:
-------------------------------------
I'll take a look tonight.
> Add input support for Hadoop Streaming
> --------------------------------------
>
> Key: CASSANDRA-1497
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1497
> Project: Cassandra
> Issue Type: New Feature
> Components: Hadoop
> Reporter: Jeremy Hanna
> Assignee: Jeremy Hanna
> Fix For: 0.7.1
>
> Attachments: 0001-An-updated-avro-based-input-streaming-solution.patch
>
>
> related to CASSANDRA-1368 - create similar functionality for input streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.