You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jeremy Hanna (JIRA)" <ji...@apache.org> on 2011/04/21 17:58:05 UTC
[jira] [Created] (CASSANDRA-2527) Add ability to snapshot data as
input to hadoop jobs
Add ability to snapshot data as input to hadoop jobs
----------------------------------------------------
Key: CASSANDRA-2527
URL: https://issues.apache.org/jira/browse/CASSANDRA-2527
Project: Cassandra
Issue Type: Improvement
Reporter: Jeremy Hanna
It is desirable to have immutable inputs to hadoop jobs for the duration of the job. That way re-execution of individual tasks do not alter the output. One way to accomplish this would be to snapshot the data that is used as input to a job.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2527) Add ability to snapshot data as
input to hadoop jobs
Posted by "Wojciech Meler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059356#comment-13059356 ]
Wojciech Meler commented on CASSANDRA-2527:
-------------------------------------------
It would be great to have more generic client access to snapshot data. Maybe snapshots should be visible as new keyspaces? Or maybe we should throw away snapshots and start cloning keyspaces? If cloned keyspace could be read-only it would work out of the box :).
> Add ability to snapshot data as input to hadoop jobs
> ----------------------------------------------------
>
> Key: CASSANDRA-2527
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2527
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jeremy Hanna
> Labels: hadoop
>
> It is desirable to have immutable inputs to hadoop jobs for the duration of the job. That way re-execution of individual tasks do not alter the output. One way to accomplish this would be to snapshot the data that is used as input to a job.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2527) Add ability to snapshot data as
input to hadoop jobs
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023391#comment-13023391 ]
Jeremy Hanna commented on CASSANDRA-2527:
-----------------------------------------
I remember the acunu guys talking about having point-in-time versioning for sstables. That would be a helpful feature for doing something like this.
> Add ability to snapshot data as input to hadoop jobs
> ----------------------------------------------------
>
> Key: CASSANDRA-2527
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2527
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jeremy Hanna
> Labels: hadoop
>
> It is desirable to have immutable inputs to hadoop jobs for the duration of the job. That way re-execution of individual tasks do not alter the output. One way to accomplish this would be to snapshot the data that is used as input to a job.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2527) Add ability to snapshot data as
input to hadoop jobs
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2527:
--------------------------------------
Fix Version/s: 1.2
Issue Type: New Feature (was: Improvement)
> Add ability to snapshot data as input to hadoop jobs
> ----------------------------------------------------
>
> Key: CASSANDRA-2527
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2527
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jeremy Hanna
> Labels: hadoop
> Fix For: 1.2
>
>
> It is desirable to have immutable inputs to hadoop jobs for the duration of the job. That way re-execution of individual tasks do not alter the output. One way to accomplish this would be to snapshot the data that is used as input to a job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2527) Add ability to snapshot data as
input to hadoop jobs
Posted by "Ilya Maykov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276166#comment-13276166 ]
Ilya Maykov commented on CASSANDRA-2527:
----------------------------------------
We wrote a Hadoop InputFormat class that could read SSTable files directly, completely bypassing the Cassandra server - not that hard to do as the SSTable file format is pretty simple. Then we exported the snapshot directories over NFS to our hadoop workers and ran the MR job that way. Obviously only useful if you want to iterate through all of the data in your Cassandra cluster. Also has a lot of overhead - this approach reads through stale versions of data that haven't been compacted away yet, and reads RF replicas of each row ... exposing snapshots in special snapshot keyspaces so they could be mapped using stock hadoop mappers may be a better way to go.
> Add ability to snapshot data as input to hadoop jobs
> ----------------------------------------------------
>
> Key: CASSANDRA-2527
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2527
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jeremy Hanna
> Labels: hadoop
>
> It is desirable to have immutable inputs to hadoop jobs for the duration of the job. That way re-execution of individual tasks do not alter the output. One way to accomplish this would be to snapshot the data that is used as input to a job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2527) Add ability to snapshot data as
input to hadoop jobs
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-2527:
-----------------------------------------
Assignee: Brandon Williams
> Add ability to snapshot data as input to hadoop jobs
> ----------------------------------------------------
>
> Key: CASSANDRA-2527
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2527
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jeremy Hanna
> Assignee: Brandon Williams
> Labels: hadoop
> Fix For: 1.2
>
>
> It is desirable to have immutable inputs to hadoop jobs for the duration of the job. That way re-execution of individual tasks do not alter the output. One way to accomplish this would be to snapshot the data that is used as input to a job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2527) Add ability to snapshot data as
input to hadoop jobs
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276205#comment-13276205 ]
Jonathan Ellis commented on CASSANDRA-2527:
-------------------------------------------
I guess the main drawback to directly reading the sstables is, you either have to set up some kind of external NFS sharing like Ilya did, or you have to be willing to live without non-local data access.
I'd be okay with saying "yes, you can run m/r against snapshots, but only with local map tasks." Better than not having it at all...
> Add ability to snapshot data as input to hadoop jobs
> ----------------------------------------------------
>
> Key: CASSANDRA-2527
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2527
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jeremy Hanna
> Labels: hadoop
>
> It is desirable to have immutable inputs to hadoop jobs for the duration of the job. That way re-execution of individual tasks do not alter the output. One way to accomplish this would be to snapshot the data that is used as input to a job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira