You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "NIKHIL THALLAPALLI (JIRA)" <ji...@apache.org> on 2018/08/15 14:14:00 UTC

[jira] [Updated] (CASSANDRA-14643) Performance overhead with COPY command while bulk data loading

     [ https://issues.apache.org/jira/browse/CASSANDRA-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

NIKHIL THALLAPALLI updated CASSANDRA-14643:
-------------------------------------------
    Description: 
Hello Team,

We are facing performance overhead with COPY utility of cql while doing bulk loading of data.

It took approximately 6 hours while loading around 39L records through COPY command.

 

Please be noted that the frozen types used in the above table structure contains around 15 sets each.

so, please let us know the techniques of optimization to achieve faster loading of data, And also suggest if any parameters can be reset to increase the efficiency.

Your prompt response on this would be highly appreciated

 

Thank you,

Nikhil

  was:
Hello Team,

We are facing performance overhead with COPY utility of cql while doing bulk loading of data.

It took approximately 6 hours while loading around 39L records through COPY command.

 

Table structure used for bulk data loading.

CREATE TABLE linemacro (

subscriber_id text,
 ocs_reservation_key text,
 session_fragment_key text,
 balance_adj_fragment frozen<balance_adjustment_fragment>,
 consumption_msg_sent_ind int,
 dynamic_alloc_chrg_resrv_key text,
 dynamic_alloc_count int,
 dynamic_alloc_count_pending int,
 dynamic_discount_profile frozen<dynamic_discount>,
 dynamic_otc_data frozen<one_time_charge>,
 last_update_dtm timestamp,
 logically_exhausted_ind int,
 low_balance_ind int,
 macro_balance_summary frozen<line_macro_summary>,
 macro_profile frozen<line_macro_profile>,
 offline_charges_summary frozen<offline_charges_summary>,
 offline_version_num bigint,
 schema_ver_num int,
 total_quota_reset bigint,
 ver_num bigint,
 PRIMARY KEY (subscriber_id, ocs_reservation_key, session_fragment_key)
) WITH CLUSTERING ORDER BY (ocs_reservation_key ASC, session_fragment_key ASC)
 AND bloom_filter_fp_chance = 0.1
 AND caching = \{'keys': 'ALL', 'rows_per_partition': 'NONE'}
 AND comment = ''
 AND compaction = \{'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
 AND compression = \{'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
 AND crc_check_chance = 1.0
 AND dclocal_read_repair_chance = 0.1
 AND default_time_to_live = 0
 AND gc_grace_seconds = 0
 AND max_index_interval = 2048
 AND memtable_flush_period_in_ms = 0
 AND min_index_interval = 128
 AND read_repair_chance = 0.0
 AND speculative_retry = '99PERCENTILE';

 

Please be noted that the frozen types used in the above table structure contains around 15 sets each.

so, please let us know the techniques of optimization to achieve faster loading of data, And also suggest if any parameters can be reset to increase the efficiency.

Your prompt response on this would be highly appreciated

 

Thank you,

Nikhil


> Performance overhead with COPY command while bulk data loading
> --------------------------------------------------------------
>
>                 Key: CASSANDRA-14643
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14643
>             Project: Cassandra
>          Issue Type: Task
>          Components: CQL
>            Reporter: NIKHIL THALLAPALLI
>            Priority: Major
>
> Hello Team,
> We are facing performance overhead with COPY utility of cql while doing bulk loading of data.
> It took approximately 6 hours while loading around 39L records through COPY command.
>  
> Please be noted that the frozen types used in the above table structure contains around 15 sets each.
> so, please let us know the techniques of optimization to achieve faster loading of data, And also suggest if any parameters can be reset to increase the efficiency.
> Your prompt response on this would be highly appreciated
>  
> Thank you,
> Nikhil



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org