You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by onmstester onmstester <on...@zoho.com> on 2018/04/08 09:15:40 UTC

copy from one table to another

Is there any way to copy some part of a table to another table in cassandra? A large amount of data should be copied so i don't want to fetch data to client and stream it back to cassandra using cql.



Sent using Zoho Mail

Re: copy from one table to another

Posted by Christophe Schmitz <ch...@instaclustr.com>.

If you need this kind of logic, you might want to consider using Spark.
It's often used for data migration.
You could load your list of partition_key in a Spark RDD, then
use joinWithCassandraTable, and write the result back to your destination
table.
Just before the join, you could use repartitionByCassandraReplica on your
RDD to have better data locality.
This documentation can be helpful:
https://github.com/datastax/spark-cassandra-connector/blob/master/doc/2_loading.md#performing-efficient-joins-with-cassandra-tables-since-12

Hope it helps

Cheers,
Christophe

On 9 April 2018 at 13:09, onmstester onmstester <on...@zoho.com> wrote:

> Thank you all
> I need something like this:
> insert into table test2 select * from test1 where
> partition_key='SOME_KEYS';
> The problem with copying sstable is that original table contains some
> billions of records and i only want some hundred millions of records from
> the table, so after copy/pasting big sstables in so many nodes i should
> wait for a deletion that would take so long to response:
> delete from test2 where partition_key != 'SOME_KEYS'
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
> ---- On Mon, 09 Apr 2018 06:14:02 +0430 *Dmitry Saprykin
> <saprykin.dmitry@gmail.com <sa...@gmail.com>>* wrote ----
>
> IMHO The best step by step description of what you need to do is here
>
> https://issues.apache.org/jira/browse/CASSANDRA-1585?
> focusedCommentId=13488959&page=com.atlassian.jira.
> plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13488959
>
> The only difference is that you need to copy data from one table only. I
> did it for a whole keyspace.
>
>
>
>
> On Sun, Apr 8, 2018 at 3:06 PM Jean Carlo <je...@gmail.com>
> wrote:
>
> You can use the same procedure to restore a table from snapshot from
> datastax webpage
>
> https://docs.datastax.com/en/cassandra/2.1/cassandra/
> operations/ops_backup_snapshot_restore_t.html
> Just two modifications.
>
> after step 5, modify the name of the sstables to add the name of the table
> you want to copy to.
>
> and in the step 6 copy the sstables to the right directory corresponding
> to the tale you want to copy to.
>
> Be sure you have an snapshot of the table source and ignore step 4 of
> course
>
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
> On Sun, Apr 8, 2018 at 6:33 PM, Dmitry Saprykin <saprykin.dmitry@gmail.com
> > wrote:
>
> You can copy hardlinks to ALL SSTables from old to new table and then
> delete part of data you do not need in a new one.
>
> On Sun, Apr 8, 2018 at 10:20 AM, Nitan Kainth <ni...@gmail.com>
> wrote:
>
> If it for testing and you don’t need any specific data, just copy a set of
> sstables with all files of that sequence and move to target tables
> directory and rename it.
>
> Restart target node or run nodetool refresh
>
> Sent from my iPhone
>
> On Apr 8, 2018, at 4:15 AM, onmstester onmstester <on...@zoho.com>
> wrote:
>
> Is there any way to copy some part of a table to another table in
> cassandra? A large amount of data should be copied so i don't want to fetch
> data to client and stream it back to cassandra using cql.
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>
>


-- 

*Christophe Schmitz - **VP Consulting*

AU: +61 4 03751980 / FR: +33 7 82022899

<https://www.facebook.com/instaclustr>   <https://twitter.com/instaclustr>
<https://www.linkedin.com/company/instaclustr>

Read our latest technical blog posts here
<https://www.instaclustr.com/blog/>. This email has been sent on behalf
of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This
email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Re: copy from one table to another

Posted by onmstester onmstester <on...@zoho.com>.

Thank you all

I need something like this:

insert into table test2 select * from test1 where partition_key='SOME_KEYS';

The problem with copying sstable is that original table contains some billions of records and i only want some hundred millions of records from the table, so after copy/pasting big sstables in so many nodes i should wait for a deletion that would take so long to response:

delete from test2 where partition_key != 'SOME_KEYS'


Sent using Zoho Mail






---- On Mon, 09 Apr 2018 06:14:02 +0430 Dmitry Saprykin &lt;saprykin.dmitry@gmail.com&gt; wrote ----




IMHO The best step by step description of what you need to do is here



https://issues.apache.org/jira/browse/CASSANDRA-1585?focusedCommentId=13488959&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13488959



The only difference is that you need to copy data from one table only. I did it for a whole keyspace.










On Sun, Apr 8, 2018 at 3:06 PM Jean Carlo &lt;jean.jeancarl48@gmail.com&gt; wrote:






You can use the same procedure to restore a table from snapshot from datastax webpage 



https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_backup_snapshot_restore_t.html


Just two modifications.




after step 5, modify the name of the sstables to add the name of the table you want to copy to.




and in the step 6 copy the sstables to the right directory corresponding to the tale you want to copy to.




Be sure you have an snapshot of the table source and ignore step 4 of course 






Saludos



Jean Carlo


"The best way to predict the future is to invent it" Alan Kay






On Sun, Apr 8, 2018 at 6:33 PM, Dmitry Saprykin &lt;saprykin.dmitry@gmail.com&gt; wrote:

You can copy hardlinks to ALL SSTables from old to new table and then delete part of data you do not need in a new one.



On Sun, Apr 8, 2018 at 10:20 AM, Nitan Kainth &lt;nitankainth@gmail.com&gt; wrote:

If it for testing and you don’t need any specific data, just copy a set of sstables with all files of that sequence and move to target tables directory and rename it.



Restart target node or run nodetool refresh 



Sent from my iPhone



On Apr 8, 2018, at 4:15 AM, onmstester onmstester &lt;onmstester@zoho.com&gt; wrote:


Is there any way to copy some part of a table to another table in cassandra? A large amount of data should be copied so i don't want to fetch data to client and stream it back to cassandra using cql.



Sent using Zoho Mail

Re: copy from one table to another

Posted by Dmitry Saprykin <sa...@gmail.com>.

IMHO The best step by step description of what you need to do is here

https://issues.apache.org/jira/browse/CASSANDRA-1585?focusedCommentId=13488959&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13488959

The only difference is that you need to copy data from one table only. I
did it for a whole keyspace.




On Sun, Apr 8, 2018 at 3:06 PM Jean Carlo <je...@gmail.com> wrote:

> You can use the same procedure to restore a table from snapshot from
> datastax webpage
>
>
> https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_backup_snapshot_restore_t.html
>
> Just two modifications.
>
> after step 5, modify the name of the sstables to add the name of the table
> you want to copy to.
>
> and in the step 6 copy the sstables to the right directory corresponding
> to the tale you want to copy to.
>
>
> Be sure you have an snapshot of the table source and ignore step 4 of
> course
>
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
> On Sun, Apr 8, 2018 at 6:33 PM, Dmitry Saprykin <saprykin.dmitry@gmail.com
> > wrote:
>
>> You can copy hardlinks to ALL SSTables from old to new table and then
>> delete part of data you do not need in a new one.
>>
>> On Sun, Apr 8, 2018 at 10:20 AM, Nitan Kainth <ni...@gmail.com>
>> wrote:
>>
>>> If it for testing and you don’t need any specific data, just copy a set
>>> of sstables with all files of that sequence and move to target tables
>>> directory and rename it.
>>>
>>> Restart target node or run nodetool refresh
>>>
>>> Sent from my iPhone
>>>
>>> On Apr 8, 2018, at 4:15 AM, onmstester onmstester <on...@zoho.com>
>>> wrote:
>>>
>>> Is there any way to copy some part of a table to another table in
>>> cassandra? A large amount of data should be copied so i don't want to fetch
>>> data to client and stream it back to cassandra using cql.
>>>
>>> Sent using Zoho Mail <https://www.zoho.com/mail/>
>>>
>>>
>>>
>>
>