You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Vijay (Created) (JIRA)" <ji...@apache.org> on 2012/01/03 01:40:20 UTC
[jira] [Created] (CASSANDRA-3690) Streaming CommitLog backup
Streaming CommitLog backup
--------------------------
Key: CASSANDRA-3690
URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
Project: Cassandra
Issue Type: Bug
Components: Tools
Affects Versions: 1.1
Reporter: Vijay
Assignee: Vijay
Priority: Minor
Fix For: 1.1
Problems with the current SST backups
1) The current backup doesn't allow us to restore point in time (within a SST)
2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
For some use cases where there is less writes it becomes increasingly difficult to time it right.
4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
Disadvantages of the new solution
1) Over head in processing the mutations during the recover phase.
2) More complicated solution than just copying the file to the archive.
Additional advantages:
Online and offline restore.
Close to live incremental backup.
Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
There are 3 Options in the initial implementation:
1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
3) Restore -> this will get the serialized bytes and apply's the mutation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3690:
-----------------------------
Attachment: 0001-CASSANDRA-3690-v4.patch
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-CASSANDRA-3690-v4.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228958#comment-13228958 ]
Jonathan Ellis commented on CASSANDRA-3690:
-------------------------------------------
I don't see any uses of CommitLogRecover outside of CommitLog, which makes me think this is an unrelated refactoring. Is that correct? If so, let's split that out of this ticket.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3690:
--------------------------------------
Attachment: 3690-v6.txt
v6 attached. The primary changes made are fixes to the Future logic: the only way you'll get a null Future back is if no archive tack was submitted; if it errors out, you'll get an ExecutionException when you call get(), but never a null.
Updated getArchivingSegmentNames javadoc to emphasize that it does NOT include failed archive attempts. Not sure if this is what was intended.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-CASSANDRA-3690-v4.patch, 0001-CASSANDRA-3690-v5.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch, 3690-v6.txt
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3690:
-----------------------------
Attachment: 0001-CASSANDRA-3690-v5.patch
Hi Jonathan,
v5 removes the recycle related changes and
Added 2 JMX (getActiveSegmentNames and getArchivingSegmentNames)
(list all files) - (getActiveSegmentNames) will provide a view of orphan files which failed archiving...
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-CASSANDRA-3690-v4.patch, 0001-CASSANDRA-3690-v5.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3690) Streaming CommitLog
backup
Posted by "Vijay (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191899#comment-13191899 ]
Vijay edited comment on CASSANDRA-3690 at 1/24/12 5:43 AM:
-----------------------------------------------------------
0001 => Adds a configuration so we can avoid recycling in case some one wants to copy the files across to another location like a archive logs
0002 => Adds CommitLogListener, implementation can recive the updates to the commitlogs.
0003 => helper JMX in case the user wants to query the active CL's
0004 => this can go to the tools folder/we dont need to commit it to the core.
was (Author: vijay2win@yahoo.com):
0001 => Adds CommitLogListener, implementation can recive the updates to the commitlogs. This also adds a configuration so we can avoid recycling in case some one wants to copy the files across to another location like a archive logs
0002 => helper JMX in case the user wants to query the active CL's
0003 => this can go to the tools folder/we dont need to commit it to the core.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 1.1
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249119#comment-13249119 ]
Jonathan Ellis commented on CASSANDRA-3690:
-------------------------------------------
bq. it will good to have unique names in the backup sometimes so we dont overwrite
If you think about it, "target filename" is just a suggestion... you'd be free to ignore it and generate a different filename (incorporating timestamp for instance, or even a uuid) in the archive script.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-CASSANDRA-3690-v4.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3690) Streaming CommitLog
backup
Posted by "Vijay (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225656#comment-13225656 ]
Vijay edited comment on CASSANDRA-3690 at 3/8/12 11:12 PM:
-----------------------------------------------------------
Hi Jonathan,
Attached patch does exactly what we discussed here. Its almost the same as PostgreSQL :)
In addition we can start the node with -Dcassandra.join_ring=false and then use JMX to restore files one by one via JMX.
Plz let me know.
was (Author: vijay2win@yahoo.com):
Hi Jonathan,
Attached patch does exactly what we discussed here. Its almost the same as Postgress :)
In addition we can start the node with -Dcassandra.join_ring=false and then use JMX to restore files one by one via JMX.
Plz let me know.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249126#comment-13249126 ]
Vijay commented on CASSANDRA-3690:
----------------------------------
Hi Jonathan, Agree, would you like to remove the configuration to disable recycle (which this patch added)? let me know... Thanks!
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-CASSANDRA-3690-v4.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3690:
-----------------------------
Attachment: 0001-Make-commitlog-recycle-configurable.patch
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 1.1
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-Make-commitlog-recycle-configurable.patch, 0001-support-commit-log-listener.patch, 0002-helper-jmx-methods.patch, 0003-external-commitlog-with-sockets.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3690:
-----------------------------
Attachment: 0003-external-commitlog-with-sockets.patch
0002-helper-jmx-methods.patch
0001-support-commit-log-listener.patch
0001 => Adds CommitLogListener, implementation can recive the updates to the commitlogs. This also adds a configuration so we can avoid recycling in case some one wants to copy the files across to another location like a archive logs
0002 => helper JMX in case the user wants to query the active CL's
0003 => this can go to the tools folder/we dont need to commit it to the core.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 1.1
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-support-commit-log-listener.patch, 0002-helper-jmx-methods.patch, 0003-external-commitlog-with-sockets.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3690:
-----------------------------
Attachment: (was: 0001-Make-commitlog-recycle-configurable.patch)
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 1.1
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205043#comment-13205043 ]
Jonathan Ellis commented on CASSANDRA-3690:
-------------------------------------------
bq. But there is additional IO which the server has to do to copy the archive logs to a different location (not locally)... While streaming the Commit log back to the server we have to copy it first and then read it back which is also a over head in recovery.
What if you mounted the archive logs via s3fs?
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213995#comment-13213995 ]
Jonathan Ellis commented on CASSANDRA-3690:
-------------------------------------------
I'm skeptical of trying to do this on top of triggers. First, CASSANDRA-1311 seems to lean towards coordinator-level triggers rather than replica-level. Second, I don't think it makes sense for a Trigger-level API to deal with raw bytes, which would mean losing efficiency from having to re-serialize RowMutations.
I like the postgresql approach: http://www.postgresql.org/docs/9.1/static/continuous-archiving.html -- briefly, you configure an {{archive_command}} that tells it how you want it to copy full log segments off-server when full, and set up a recovery.conf file when you want to recover, which includes a {{restore_command}} that is the inverse of the archive command.
The main difference is that postgresql's default wal segment size is 16MB, which gives them a finer resolution than our 128MB. I can't think of a reason we can't lower ours, though.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3690:
-----------------------------
Attachment: (was: 0001-support-commit-log-listener.patch)
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 1.1
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213931#comment-13213931 ]
Vijay commented on CASSANDRA-3690:
----------------------------------
Starting to think... What we really want is a Async Triggers CASSANDRA-1311 which listens for all the updates + a way to restore the data with mutation before starting the node. In someways thats what the original patch was trying to do will it make sense to merge these two efforts?
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180954#comment-13180954 ]
Jonathan Ellis commented on CASSANDRA-3690:
-------------------------------------------
The socket business sounds complicated. CASSANDRA-1602 is a lot more straightforward, I'd recommend starting with that.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 1.1
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249329#comment-13249329 ]
Jonathan Ellis commented on CASSANDRA-3690:
-------------------------------------------
I do think we should make recycling always-on; it's a non-negligible performance win and so far we don't have a use case that requires disabling it.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-CASSANDRA-3690-v4.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3690:
-----------------------------
Description:
Problems with the current SST backups
1) The current backup doesn't allow us to restore point in time (within a SST)
2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
For some use cases where there is less writes it becomes increasingly difficult to time it right.
4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
Disadvantages of the new solution
1) Over head in processing the mutations during the recover phase.
2) More complicated solution than just copying the file to the archive.
Additional advantages:
Online and offline restore.
Close to live incremental backup.
Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
There are 3 Options in the initial implementation:
1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
3) Restore -> this will get the serialized bytes and apply's the mutation.
Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
was:
Problems with the current SST backups
1) The current backup doesn't allow us to restore point in time (within a SST)
2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
For some use cases where there is less writes it becomes increasingly difficult to time it right.
4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
Disadvantages of the new solution
1) Over head in processing the mutations during the recover phase.
2) More complicated solution than just copying the file to the archive.
Additional advantages:
Online and offline restore.
Close to live incremental backup.
Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
There are 3 Options in the initial implementation:
1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
3) Restore -> this will get the serialized bytes and apply's the mutation.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 1.1
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180959#comment-13180959 ]
Vijay commented on CASSANDRA-3690:
----------------------------------
Hi Jonathan, But there is additional IO which the server has to do to copy the archive logs to a different location (not locally)...
While streaming the Commit log back to the server we have to copy it first and then read it back which is also a over head in recovery.
Something like copying the data to S3 in amazon and copying right back for the node for recovery. (this backup will also be used for test cluster refresh for prod data and BI which is completely a different system)
Recovery in most case are loose of instance or the whole cluster (Virtual machines).
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 1.1
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3690:
-----------------------------
Attachment: (was: 0001-CASSANDRA-3690.patch)
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-CASSANDRA-3690-v4.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Jason Brown (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213096#comment-13213096 ]
Jason Brown commented on CASSANDRA-3690:
----------------------------------------
We spent some time looking into s3fs, and ran into problems between fuse (which s3fs depends on) and mmap. I put together a small java app to simulate writing to local disc vs. s3fs using mmapp'ed and non-mmap'ed files. (Note most of my java sample code is based on CommitLogSegment's implementation.) We found the non-mmap'ed files wrote to s3fs without a hitch, but writing to the mmap'ed S3 mount failed. The failure was different between OpenJDK 6 vs. Sun JDK, but they were both SIGBUS errors. Also, I ran the tests on my local ubuntu box running Linux 3.0.0-12, fuse version (fusermount --version) at 2.8.4.
At this point, it seems like s3fs isn't as viable as one would hope.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3690:
-----------------------------
Attachment: (was: 0002-helper-jmx-methods.patch)
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 1.1
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247729#comment-13247729 ]
Jonathan Ellis commented on CASSANDRA-3690:
-------------------------------------------
I've made a bunch of minor changes and pushed to https://github.com/jbellis/cassandra/branches/3690-v3.
I noticed that we need to wait for the archive to finish whether we end up recycling or not. Seems to me it would be simpler to continue to always recycle, but (as we have here) wait for the archive first. So archive can copy off to s3 or whatever directly, instead of ln somewhere else as an intermediate step. Total i/o will be lower and commitlog will create extra segments if needed in the meantime.
Maybe we should also have a restore_list_segments command as well, so we can query s3 (again for instance) directly and have restore_command pull from there, rather than requiring a local directory?
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3690:
-----------------------------
Attachment: 0001-CASSANDRA-3690.patch
Hi Jonathan, Attached patch incorporates all the recommended changes.... except
>>> Maybe we should also have a restore_list_segments command as well, so we can query s3 (again for instance) directly and have restore_command pull from there, rather than requiring a local directory?
IMHO. It might be better if we have a streaming API to list and stream the data in... otherwise we need have to download to the local FS anyways, So it will be better to incrementally download and use the JMX to restore the files independently (example: A external agent), that may be a simple solution for now..... If the user has a NFS mount it will work even better all he needs to do is to "ln -s" location and he is done :)
Plz note that i also removed the requirement to turn off recycling for backup (as recommended), but i left that as configurable because it will good to have unique names in the backup sometimes so we dont overwrite :)
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-CASSANDRA-3690.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Jonathan Ellis (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-3690.
---------------------------------------
Resolution: Fixed
committed w/ some final improvements to yaml comments
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-CASSANDRA-3690-v4.patch, 0001-CASSANDRA-3690-v5.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch, 3690-v6.txt
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213911#comment-13213911 ]
Jonathan Ellis commented on CASSANDRA-3690:
-------------------------------------------
That does rule out using s3fs in read/write mode, but I imagine that would be a pretty bad idea from a latency standpoint anyway. But in the context of just mounting log files for replay/recover, CommitLog uses RandomAccessReader which is ordinary buffered i/o.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3690:
-----------------------------
Attachment: 0001-CASSANDRA-3690-v2.patch
Hi Jonathan,
Attached patch does exactly what we discussed here. Its almost the same as Postgress :)
In addition we can start the node with -Dcassandra.join_ring=false and then use JMX to restore files one by one via JMX.
Plz let me know.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3690:
-----------------------------
Attachment: (was: 0003-external-commitlog-with-sockets.patch)
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 1.1
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3690:
-----------------------------
Attachment: 0005-cmmiting-comments-to-yaml.patch
0004-external-commitlog-with-sockets.patch
0003-helper-jmx-methods.patch
0002-support-commit-log-listener.patch
0001-Make-commitlog-recycle-configurable.patch
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 1.1
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228962#comment-13228962 ]
Vijay commented on CASSANDRA-3690:
----------------------------------
Hi Jonathan, The refactor is mainly for the following:
>>> In addition we can start the node with -Dcassandra.join_ring=false and then use JMX to restore files one by one via JMX.
This allows us to start the recovery process before all the files are downloaded from S3/External source. I can do that it in another ticket, let me know. Thanks!
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249876#comment-13249876 ]
Vijay commented on CASSANDRA-3690:
----------------------------------
+1
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-CASSANDRA-3690-v4.patch, 0001-CASSANDRA-3690-v5.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch, 3690-v6.txt
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3690) Streaming CommitLog backup
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247738#comment-13247738 ]
Jonathan Ellis commented on CASSANDRA-3690:
-------------------------------------------
Also: would be nice to get rid of the new Thread / busywait archive dance. If we used an ExecutorService instead, we could add the Future to the segment and just say segment.waitForArchive(), no looping.
> Streaming CommitLog backup
> --------------------------
>
> Key: CASSANDRA-3690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3690
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3690-v2.patch, 0001-Make-commitlog-recycle-configurable.patch, 0002-support-commit-log-listener.patch, 0003-helper-jmx-methods.patch, 0004-external-commitlog-with-sockets.patch, 0005-cmmiting-comments-to-yaml.patch
>
>
> Problems with the current SST backups
> 1) The current backup doesn't allow us to restore point in time (within a SST)
> 2) Current SST implementation needs the backup to read from the filesystem and hence additional IO during the normal operational Disks
> 3) in 1.0 we have removed the flush interval and size when the flush will be triggered per CF,
> For some use cases where there is less writes it becomes increasingly difficult to time it right.
> 4) Use cases which needs BI which are external (Non cassandra), needs the data in regular intervals than waiting for longer or unpredictable intervals.
> Disadvantages of the new solution
> 1) Over head in processing the mutations during the recover phase.
> 2) More complicated solution than just copying the file to the archive.
> Additional advantages:
> Online and offline restore.
> Close to live incremental backup.
> Note: If the listener agent gets restarted, it is the agents responsibility to Stream the files missed or incomplete.
> There are 3 Options in the initial implementation:
> 1) Backup -> Once a socket is connected we will switch the commit log and send new updates via the socket.
> 2) Stream -> will take the absolute path of the file and will read the file and send the updates via the socket.
> 3) Restore -> this will get the serialized bytes and apply's the mutation.
> Side NOTE: (Not related to this patch as such) The agent which will take incremental backup is planned to be open sourced soon (Name: Priam).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira