You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by GitBox <gi...@apache.org> on 2020/01/07 01:43:10 UTC
[GitHub] [cassandra] dvohra opened a new pull request #402: New/virtualtables
dvohra opened a new pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402
Added a page on new feature Virtual Tables.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379571223
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the
+virtual table presently. A total of 224 settings get listed presently.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -----------------------------------------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_audit_logs_dir |
+ /home/ec2-user/cassandra/logs/audit/
+ audit_logging_options_enabled |
+ false
+ audit_logging_options_excluded_categories |
+ audit_logging_options_excluded_keyspaces |
+ system,system_schema,system_virtual_schema
+ audit_logging_options_excluded_users |
+ audit_logging_options_included_categories |
+ audit_logging_options_included_keyspaces |
+ audit_logging_options_included_users |
+ audit_logging_options_logger |
+ BinAuditLogger
+ authenticator |
+ AllowAllAuthenticator
+ authorizer |
+ AllowAllAuthorizer
+ auto_bootstrap |
+ true
+ auto_snapshot |
+ true
+ automatic_sstable_upgrade |
+ false
+ back_pressure_enabled |
+ false
+ back_pressure_strategy |
+ org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}
+ broadcast_address |
+ null
+ broadcast_rpc_address |
+ null
+ cluster_name |
+ Test Cluster
+ column_index_cache_size_in_kb |
+ 2
+ column_index_size_in_kb |
+ 64
+ commit_failure_policy |
+ stop
+ commitlog_compression |
+ null
+ commitlog_directory |
+ /home/ec2-user/cassandra/data/commitlog
+ commitlog_max_compression_buffers_in_pool |
+ 3
+ commitlog_periodic_queue_size |
+ -1
+ commitlog_segment_size_in_mb |
+ 32
+ commitlog_sync |
+ periodic
+ commitlog_sync_batch_window_in_ms |
+ NaN
+ commitlog_sync_group_window_in_ms |
+ NaN
+ commitlog_sync_period_in_ms |
+ 10000
+ commitlog_total_space_in_mb |
+ 2556
+ compaction_large_partition_warning_threshold_mb |
+ 100
+ compaction_throughput_mb_per_sec |
+ 16
+ concurrent_compactors |
+ 2
+ concurrent_counter_writes |
+ 32
+ concurrent_materialized_view_builders |
+ 1
+ concurrent_materialized_view_writes |
+ 32
+ concurrent_reads |
+ 32
+ concurrent_replicates |
+ null
+ concurrent_validations |
+ 2147483647
+ concurrent_writes |
+ 32
+ credentials_update_interval_in_ms |
+ -1
+ credentials_validity_in_ms |
+ 2000
+ cross_node_timeout |
+ false
+ data_file_directories |
+ [/home/ec2-user/cassandra/data/data]
+ diagnostic_events_enabled |
+ false
+ disk_access_mode |
+ mmap
+ disk_failure_policy |
+ stop
+ enable_materialized_views |
+ false
+ enable_sasi_indexes |
+ false
+ enable_scripted_user_defined_functions |
+ false
+ enable_transient_replication |
+ false
+ enable_user_defined_functions |
+ false
+ enable_user_defined_functions_threads |
+ true
+ endpoint_snitch |
+ SimpleSnitch
+ file_cache_round_up |
+ false
+ file_cache_size_in_mb |
+ 251
+ full_query_logging_options |
+ FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true,
+ max_queue_weight=268435456, max_log_size=17179869184}
+ gc_log_threshold_in_ms |
+ 200
+ gc_warn_threshold_in_ms |
+ 1000
+ hinted_handoff_disabled_datacenters |
+ []
+ hinted_handoff_enabled |
+ true
+ hinted_handoff_throttle_in_kb |
+ 1024
+ hints_compression |
+ null
+ hints_directory |
+ /home/ec2-user/cassandra/data/hints
+ hints_flush_period_in_ms |
+ 10000
+ ideal_consistency_level |
+ null
+ incremental_backups |
+ false
+ initial_token |
+ null
+ inter_dc_stream_throughput_outbound_megabits_per_sec |
+ 200
+ inter_dc_tcp_nodelay |
+ false
+ internode_application_receive_queue_capacity_in_bytes |
+ 4194304
+ internode_application_receive_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+ internode_application_receive_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_application_send_queue_capacity_in_bytes |
+ 4194304
+ internode_application_send_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+
+
+ internode_application_send_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_authenticator |
+ null
+ internode_compression |
+ dc
+ listen_address |
+ localhost
+ listen_interface |
+ null
+ listen_on_broadcast_address |
+ false
+ max_concurrent_automatic_sstable_upgrades |
+ 1
+ max_hint_window_in_ms |
+ 10800000
+ memtable_allocation_type |
+ heap_buffers
+ min_free_space_per_drive_in_mb |
+ 50
+ native_transport_allow_older_protocols |
+ true
+ native_transport_flush_in_batches_legacy |
+ false
+ native_transport_frame_block_size_in_kb | 32
+ native_transport_idle_timeout_in_ms | 0
+ native_transport_max_concurrent_connections | -1
+ native_transport_max_concurrent_connections_per_ip | -1
+ native_transport_max_concurrent_requests_in_bytes | 105277030
+ native_transport_max_concurrent_requests_in_bytes_per_ip | 26319257
+ native_transport_max_frame_size_in_mb | 256
+ native_transport_max_threads | 128
+ native_transport_port | 9042
+ native_transport_port_ssl | null
+ network_authorizer | AllowAllNetworkAuthorizer
+ num_tokens | 256
+ partitioner | org.apache.cassandra.dht.Murmur3Partitioner
+ prepared_statements_cache_size_mb | null
+ range_request_timeout_in_ms | 10000
+ read_request_timeout_in_ms | 5000
+ repair_command_pool_full_strategy | queue
+ repair_command_pool_size | 2147483647
+ repair_session_max_tree_depth | 20
+ repair_session_space_in_mb | 62
+ repaired_data_tracking_for_partition_reads_enabled | false
+ repaired_data_tracking_for_range_reads_enabled | false
+ report_unconfirmed_repaired_data_mismatches | false
+ request_timeout_in_ms | 10000
+ role_manager | CassandraRoleManager
+ rpc_address | localhost
+ rpc_interface | null
+ saved_caches_directory | /home/ec2-user/cassandra/data/saved_caches
+ seed_provider | org.apache.cassandra.locator.SimpleSeedProvider{seeds=127.0.0.1:7000}
+ server_encryption_options_algorithm | null
+ server_encryption_options_protocol | TLS
+ slow_query_log_timeout_in_ms | 500
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ transparent_data_encryption_options_chunk_length_kb | 64
+ transparent_data_encryption_options_cipher | AES/CBC/PKCS5Padding
+ transparent_data_encryption_options_enabled | false
+ transparent_data_encryption_options_iv_length |
+ 16
+ (224 rows)
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+As no sstable tasks are running initially the ``system_views.sstable_tasks`` table lists 0 rows.
Review comment:
Added example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379569957
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
Review comment:
I have kept the statement so that sections don't appear out of context or in varying contexts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379498403
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the
+virtual table presently. A total of 224 settings get listed presently.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -----------------------------------------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_audit_logs_dir |
+ /home/ec2-user/cassandra/logs/audit/
+ audit_logging_options_enabled |
+ false
+ audit_logging_options_excluded_categories |
+ audit_logging_options_excluded_keyspaces |
+ system,system_schema,system_virtual_schema
+ audit_logging_options_excluded_users |
+ audit_logging_options_included_categories |
+ audit_logging_options_included_keyspaces |
+ audit_logging_options_included_users |
+ audit_logging_options_logger |
+ BinAuditLogger
+ authenticator |
+ AllowAllAuthenticator
+ authorizer |
+ AllowAllAuthorizer
+ auto_bootstrap |
+ true
+ auto_snapshot |
+ true
+ automatic_sstable_upgrade |
+ false
+ back_pressure_enabled |
+ false
+ back_pressure_strategy |
+ org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}
+ broadcast_address |
+ null
+ broadcast_rpc_address |
+ null
+ cluster_name |
+ Test Cluster
+ column_index_cache_size_in_kb |
+ 2
+ column_index_size_in_kb |
+ 64
+ commit_failure_policy |
+ stop
+ commitlog_compression |
+ null
+ commitlog_directory |
+ /home/ec2-user/cassandra/data/commitlog
+ commitlog_max_compression_buffers_in_pool |
+ 3
+ commitlog_periodic_queue_size |
+ -1
+ commitlog_segment_size_in_mb |
+ 32
+ commitlog_sync |
+ periodic
+ commitlog_sync_batch_window_in_ms |
+ NaN
+ commitlog_sync_group_window_in_ms |
+ NaN
+ commitlog_sync_period_in_ms |
+ 10000
+ commitlog_total_space_in_mb |
+ 2556
+ compaction_large_partition_warning_threshold_mb |
+ 100
+ compaction_throughput_mb_per_sec |
+ 16
+ concurrent_compactors |
+ 2
+ concurrent_counter_writes |
+ 32
+ concurrent_materialized_view_builders |
+ 1
+ concurrent_materialized_view_writes |
+ 32
+ concurrent_reads |
+ 32
+ concurrent_replicates |
+ null
+ concurrent_validations |
+ 2147483647
+ concurrent_writes |
+ 32
+ credentials_update_interval_in_ms |
+ -1
+ credentials_validity_in_ms |
+ 2000
+ cross_node_timeout |
+ false
+ data_file_directories |
+ [/home/ec2-user/cassandra/data/data]
+ diagnostic_events_enabled |
+ false
+ disk_access_mode |
+ mmap
+ disk_failure_policy |
+ stop
+ enable_materialized_views |
+ false
+ enable_sasi_indexes |
+ false
+ enable_scripted_user_defined_functions |
+ false
+ enable_transient_replication |
+ false
+ enable_user_defined_functions |
+ false
+ enable_user_defined_functions_threads |
+ true
+ endpoint_snitch |
+ SimpleSnitch
+ file_cache_round_up |
+ false
+ file_cache_size_in_mb |
+ 251
+ full_query_logging_options |
+ FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true,
+ max_queue_weight=268435456, max_log_size=17179869184}
+ gc_log_threshold_in_ms |
+ 200
+ gc_warn_threshold_in_ms |
+ 1000
+ hinted_handoff_disabled_datacenters |
+ []
+ hinted_handoff_enabled |
+ true
+ hinted_handoff_throttle_in_kb |
+ 1024
+ hints_compression |
+ null
+ hints_directory |
+ /home/ec2-user/cassandra/data/hints
+ hints_flush_period_in_ms |
+ 10000
+ ideal_consistency_level |
+ null
+ incremental_backups |
+ false
+ initial_token |
+ null
+ inter_dc_stream_throughput_outbound_megabits_per_sec |
+ 200
+ inter_dc_tcp_nodelay |
+ false
+ internode_application_receive_queue_capacity_in_bytes |
+ 4194304
+ internode_application_receive_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+ internode_application_receive_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_application_send_queue_capacity_in_bytes |
+ 4194304
+ internode_application_send_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+
+
+ internode_application_send_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_authenticator |
+ null
+ internode_compression |
+ dc
+ listen_address |
+ localhost
+ listen_interface |
+ null
+ listen_on_broadcast_address |
+ false
+ max_concurrent_automatic_sstable_upgrades |
+ 1
+ max_hint_window_in_ms |
+ 10800000
+ memtable_allocation_type |
+ heap_buffers
+ min_free_space_per_drive_in_mb |
+ 50
+ native_transport_allow_older_protocols |
+ true
+ native_transport_flush_in_batches_legacy |
+ false
+ native_transport_frame_block_size_in_kb | 32
+ native_transport_idle_timeout_in_ms | 0
+ native_transport_max_concurrent_connections | -1
+ native_transport_max_concurrent_connections_per_ip | -1
+ native_transport_max_concurrent_requests_in_bytes | 105277030
+ native_transport_max_concurrent_requests_in_bytes_per_ip | 26319257
+ native_transport_max_frame_size_in_mb | 256
+ native_transport_max_threads | 128
+ native_transport_port | 9042
+ native_transport_port_ssl | null
+ network_authorizer | AllowAllNetworkAuthorizer
+ num_tokens | 256
+ partitioner | org.apache.cassandra.dht.Murmur3Partitioner
+ prepared_statements_cache_size_mb | null
+ range_request_timeout_in_ms | 10000
+ read_request_timeout_in_ms | 5000
+ repair_command_pool_full_strategy | queue
+ repair_command_pool_size | 2147483647
+ repair_session_max_tree_depth | 20
+ repair_session_space_in_mb | 62
+ repaired_data_tracking_for_partition_reads_enabled | false
+ repaired_data_tracking_for_range_reads_enabled | false
+ report_unconfirmed_repaired_data_mismatches | false
+ request_timeout_in_ms | 10000
+ role_manager | CassandraRoleManager
+ rpc_address | localhost
+ rpc_interface | null
+ saved_caches_directory | /home/ec2-user/cassandra/data/saved_caches
+ seed_provider | org.apache.cassandra.locator.SimpleSeedProvider{seeds=127.0.0.1:7000}
+ server_encryption_options_algorithm | null
+ server_encryption_options_protocol | TLS
+ slow_query_log_timeout_in_ms | 500
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ transparent_data_encryption_options_chunk_length_kb | 64
+ transparent_data_encryption_options_cipher | AES/CBC/PKCS5Padding
+ transparent_data_encryption_options_enabled | false
+ transparent_data_encryption_options_iv_length |
+ 16
+ (224 rows)
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+As no sstable tasks are running initially the ``system_views.sstable_tasks`` table lists 0 rows.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+---------+------+----------+-------+------
+
+ (0 rows)
+
+The system_virtual_schema keyspace
Review comment:
I think we can drop the entire virtual schema keyspace section here. Or at most just hand wave that its used for internal Cassandra book keeping and communications with driver. This is not something a user needs to know about
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379496245
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the
+virtual table presently. A total of 224 settings get listed presently.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -----------------------------------------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_audit_logs_dir |
+ /home/ec2-user/cassandra/logs/audit/
+ audit_logging_options_enabled |
+ false
+ audit_logging_options_excluded_categories |
+ audit_logging_options_excluded_keyspaces |
+ system,system_schema,system_virtual_schema
+ audit_logging_options_excluded_users |
+ audit_logging_options_included_categories |
+ audit_logging_options_included_keyspaces |
+ audit_logging_options_included_users |
+ audit_logging_options_logger |
+ BinAuditLogger
+ authenticator |
+ AllowAllAuthenticator
+ authorizer |
+ AllowAllAuthorizer
+ auto_bootstrap |
+ true
+ auto_snapshot |
+ true
+ automatic_sstable_upgrade |
+ false
+ back_pressure_enabled |
+ false
+ back_pressure_strategy |
+ org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}
+ broadcast_address |
+ null
+ broadcast_rpc_address |
+ null
+ cluster_name |
+ Test Cluster
+ column_index_cache_size_in_kb |
+ 2
+ column_index_size_in_kb |
+ 64
+ commit_failure_policy |
+ stop
+ commitlog_compression |
+ null
+ commitlog_directory |
+ /home/ec2-user/cassandra/data/commitlog
+ commitlog_max_compression_buffers_in_pool |
+ 3
+ commitlog_periodic_queue_size |
+ -1
+ commitlog_segment_size_in_mb |
+ 32
+ commitlog_sync |
+ periodic
+ commitlog_sync_batch_window_in_ms |
+ NaN
+ commitlog_sync_group_window_in_ms |
+ NaN
+ commitlog_sync_period_in_ms |
+ 10000
+ commitlog_total_space_in_mb |
+ 2556
+ compaction_large_partition_warning_threshold_mb |
+ 100
+ compaction_throughput_mb_per_sec |
+ 16
+ concurrent_compactors |
+ 2
+ concurrent_counter_writes |
+ 32
+ concurrent_materialized_view_builders |
+ 1
+ concurrent_materialized_view_writes |
+ 32
+ concurrent_reads |
+ 32
+ concurrent_replicates |
+ null
+ concurrent_validations |
+ 2147483647
+ concurrent_writes |
+ 32
+ credentials_update_interval_in_ms |
+ -1
+ credentials_validity_in_ms |
+ 2000
+ cross_node_timeout |
+ false
+ data_file_directories |
+ [/home/ec2-user/cassandra/data/data]
+ diagnostic_events_enabled |
+ false
+ disk_access_mode |
+ mmap
+ disk_failure_policy |
+ stop
+ enable_materialized_views |
+ false
+ enable_sasi_indexes |
+ false
+ enable_scripted_user_defined_functions |
+ false
+ enable_transient_replication |
+ false
+ enable_user_defined_functions |
+ false
+ enable_user_defined_functions_threads |
+ true
+ endpoint_snitch |
+ SimpleSnitch
+ file_cache_round_up |
+ false
+ file_cache_size_in_mb |
+ 251
+ full_query_logging_options |
+ FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true,
+ max_queue_weight=268435456, max_log_size=17179869184}
+ gc_log_threshold_in_ms |
+ 200
+ gc_warn_threshold_in_ms |
+ 1000
+ hinted_handoff_disabled_datacenters |
+ []
+ hinted_handoff_enabled |
+ true
+ hinted_handoff_throttle_in_kb |
+ 1024
+ hints_compression |
+ null
+ hints_directory |
+ /home/ec2-user/cassandra/data/hints
+ hints_flush_period_in_ms |
+ 10000
+ ideal_consistency_level |
+ null
+ incremental_backups |
+ false
+ initial_token |
+ null
+ inter_dc_stream_throughput_outbound_megabits_per_sec |
+ 200
+ inter_dc_tcp_nodelay |
+ false
+ internode_application_receive_queue_capacity_in_bytes |
+ 4194304
+ internode_application_receive_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+ internode_application_receive_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_application_send_queue_capacity_in_bytes |
+ 4194304
+ internode_application_send_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+
+
+ internode_application_send_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_authenticator |
+ null
+ internode_compression |
+ dc
+ listen_address |
+ localhost
+ listen_interface |
+ null
+ listen_on_broadcast_address |
+ false
+ max_concurrent_automatic_sstable_upgrades |
+ 1
+ max_hint_window_in_ms |
+ 10800000
+ memtable_allocation_type |
+ heap_buffers
+ min_free_space_per_drive_in_mb |
+ 50
+ native_transport_allow_older_protocols |
+ true
+ native_transport_flush_in_batches_legacy |
+ false
+ native_transport_frame_block_size_in_kb | 32
+ native_transport_idle_timeout_in_ms | 0
+ native_transport_max_concurrent_connections | -1
+ native_transport_max_concurrent_connections_per_ip | -1
+ native_transport_max_concurrent_requests_in_bytes | 105277030
+ native_transport_max_concurrent_requests_in_bytes_per_ip | 26319257
+ native_transport_max_frame_size_in_mb | 256
+ native_transport_max_threads | 128
+ native_transport_port | 9042
+ native_transport_port_ssl | null
+ network_authorizer | AllowAllNetworkAuthorizer
+ num_tokens | 256
+ partitioner | org.apache.cassandra.dht.Murmur3Partitioner
+ prepared_statements_cache_size_mb | null
+ range_request_timeout_in_ms | 10000
+ read_request_timeout_in_ms | 5000
+ repair_command_pool_full_strategy | queue
+ repair_command_pool_size | 2147483647
+ repair_session_max_tree_depth | 20
+ repair_session_space_in_mb | 62
+ repaired_data_tracking_for_partition_reads_enabled | false
+ repaired_data_tracking_for_range_reads_enabled | false
+ report_unconfirmed_repaired_data_mismatches | false
+ request_timeout_in_ms | 10000
+ role_manager | CassandraRoleManager
+ rpc_address | localhost
+ rpc_interface | null
+ saved_caches_directory | /home/ec2-user/cassandra/data/saved_caches
+ seed_provider | org.apache.cassandra.locator.SimpleSeedProvider{seeds=127.0.0.1:7000}
+ server_encryption_options_algorithm | null
+ server_encryption_options_protocol | TLS
+ slow_query_log_timeout_in_ms | 500
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ transparent_data_encryption_options_chunk_length_kb | 64
+ transparent_data_encryption_options_cipher | AES/CBC/PKCS5Padding
+ transparent_data_encryption_options_enabled | false
+ transparent_data_encryption_options_iv_length |
+ 16
+ (224 rows)
Review comment:
maybe add a ``...`` in the middle here and not list the complete set. Good chance to snip things like internode_application_receive_queue_reserve_global_capacity_in_bytes and full_query_logging_options that make the table so wide
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] Deepak-Vohra commented on a change in pull request
#402: New/virtualtables
Posted by GitBox <gi...@apache.org>.
Deepak-Vohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r380876025
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,329 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose metrics through CQL
+- Expose YAML configuration information
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only, but it is likely to change
+- Virtual tables are not replicated
+- Virtual tables are local only and non distributed
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use ``LocalPartitioner``. Since a virtual table is not replicated the partitioner sorts in order of partition keys instead of by their hash.
+- Making advanced queries with ``ALLOW FILTERING`` and aggregation functions may be used with virtual tables even though in normal tables we dont recommend it
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables.
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations initially though some of these could change such as:
+
+- Cannot alter or drop virtual keyspaces or tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces; but may run aggregate functions on select
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ coordinator_scans clients tombstones_scanned internode_inbound
+ disk_usage sstable_tasks live_scanned caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in ``system_views`` virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+Some examples of how ``clients`` can be used are:
+
+- To find applications using old incompatible versions of drivers before upgrading and with ``nodetool enableoldprotocolversions`` and ``nodetool disableoldprotocolversions`` during upgrades.
+- To identify clients sending too many requests.
+- To find if SSL is enabled during the migration to and from ssl.
+
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the current configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the virtual table presently.
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_enabled | false
+ auto_snapshot | true
+ automatic_sstable_upgrade | false
+ cluster_name | Test Cluster
+ enable_transient_replication | false
+ hinted_handoff_enabled | true
+ hints_directory | /home/ec2-user/cassandra/data/hints
+ incremental_backups | false
+ initial_token | null
+ ...
+ ...
+ ...
+ rpc_address | localhost
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ (224 rows)
+
+
+The ``settings`` table can be really useful if yaml file has been changed since startup and dont know running configuration, or to find if they have been modified via jmx/nodetool or virtual tables.
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+The ``sstable_tasks`` could be used to get information about running tasks. It lists following columns.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+---------+------+----------+-------+------
Review comment:
Added example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379567768
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
Review comment:
I have added earlier that some of these are "likely to change". Best to keep the detail as currently supported. And when any of these change the page could be updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379523924
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the
+virtual table presently. A total of 224 settings get listed presently.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -----------------------------------------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_audit_logs_dir |
+ /home/ec2-user/cassandra/logs/audit/
+ audit_logging_options_enabled |
+ false
+ audit_logging_options_excluded_categories |
+ audit_logging_options_excluded_keyspaces |
+ system,system_schema,system_virtual_schema
+ audit_logging_options_excluded_users |
+ audit_logging_options_included_categories |
+ audit_logging_options_included_keyspaces |
+ audit_logging_options_included_users |
+ audit_logging_options_logger |
+ BinAuditLogger
+ authenticator |
+ AllowAllAuthenticator
+ authorizer |
+ AllowAllAuthorizer
+ auto_bootstrap |
+ true
+ auto_snapshot |
+ true
+ automatic_sstable_upgrade |
+ false
+ back_pressure_enabled |
+ false
+ back_pressure_strategy |
+ org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}
+ broadcast_address |
+ null
+ broadcast_rpc_address |
+ null
+ cluster_name |
+ Test Cluster
+ column_index_cache_size_in_kb |
+ 2
+ column_index_size_in_kb |
+ 64
+ commit_failure_policy |
+ stop
+ commitlog_compression |
+ null
+ commitlog_directory |
+ /home/ec2-user/cassandra/data/commitlog
+ commitlog_max_compression_buffers_in_pool |
+ 3
+ commitlog_periodic_queue_size |
+ -1
+ commitlog_segment_size_in_mb |
+ 32
+ commitlog_sync |
+ periodic
+ commitlog_sync_batch_window_in_ms |
+ NaN
+ commitlog_sync_group_window_in_ms |
+ NaN
+ commitlog_sync_period_in_ms |
+ 10000
+ commitlog_total_space_in_mb |
+ 2556
+ compaction_large_partition_warning_threshold_mb |
+ 100
+ compaction_throughput_mb_per_sec |
+ 16
+ concurrent_compactors |
+ 2
+ concurrent_counter_writes |
+ 32
+ concurrent_materialized_view_builders |
+ 1
+ concurrent_materialized_view_writes |
+ 32
+ concurrent_reads |
+ 32
+ concurrent_replicates |
+ null
+ concurrent_validations |
+ 2147483647
+ concurrent_writes |
+ 32
+ credentials_update_interval_in_ms |
+ -1
+ credentials_validity_in_ms |
+ 2000
+ cross_node_timeout |
+ false
+ data_file_directories |
+ [/home/ec2-user/cassandra/data/data]
+ diagnostic_events_enabled |
+ false
+ disk_access_mode |
+ mmap
+ disk_failure_policy |
+ stop
+ enable_materialized_views |
+ false
+ enable_sasi_indexes |
+ false
+ enable_scripted_user_defined_functions |
+ false
+ enable_transient_replication |
+ false
+ enable_user_defined_functions |
+ false
+ enable_user_defined_functions_threads |
+ true
+ endpoint_snitch |
+ SimpleSnitch
+ file_cache_round_up |
+ false
+ file_cache_size_in_mb |
+ 251
+ full_query_logging_options |
+ FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true,
+ max_queue_weight=268435456, max_log_size=17179869184}
+ gc_log_threshold_in_ms |
+ 200
+ gc_warn_threshold_in_ms |
+ 1000
+ hinted_handoff_disabled_datacenters |
+ []
+ hinted_handoff_enabled |
+ true
+ hinted_handoff_throttle_in_kb |
+ 1024
+ hints_compression |
+ null
+ hints_directory |
+ /home/ec2-user/cassandra/data/hints
+ hints_flush_period_in_ms |
+ 10000
+ ideal_consistency_level |
+ null
+ incremental_backups |
+ false
+ initial_token |
+ null
+ inter_dc_stream_throughput_outbound_megabits_per_sec |
+ 200
+ inter_dc_tcp_nodelay |
+ false
+ internode_application_receive_queue_capacity_in_bytes |
+ 4194304
+ internode_application_receive_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+ internode_application_receive_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_application_send_queue_capacity_in_bytes |
+ 4194304
+ internode_application_send_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+
+
+ internode_application_send_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_authenticator |
+ null
+ internode_compression |
+ dc
+ listen_address |
+ localhost
+ listen_interface |
+ null
+ listen_on_broadcast_address |
+ false
+ max_concurrent_automatic_sstable_upgrades |
+ 1
+ max_hint_window_in_ms |
+ 10800000
+ memtable_allocation_type |
+ heap_buffers
+ min_free_space_per_drive_in_mb |
+ 50
+ native_transport_allow_older_protocols |
+ true
+ native_transport_flush_in_batches_legacy |
+ false
+ native_transport_frame_block_size_in_kb | 32
+ native_transport_idle_timeout_in_ms | 0
+ native_transport_max_concurrent_connections | -1
+ native_transport_max_concurrent_connections_per_ip | -1
+ native_transport_max_concurrent_requests_in_bytes | 105277030
+ native_transport_max_concurrent_requests_in_bytes_per_ip | 26319257
+ native_transport_max_frame_size_in_mb | 256
+ native_transport_max_threads | 128
+ native_transport_port | 9042
+ native_transport_port_ssl | null
+ network_authorizer | AllowAllNetworkAuthorizer
+ num_tokens | 256
+ partitioner | org.apache.cassandra.dht.Murmur3Partitioner
+ prepared_statements_cache_size_mb | null
+ range_request_timeout_in_ms | 10000
+ read_request_timeout_in_ms | 5000
+ repair_command_pool_full_strategy | queue
+ repair_command_pool_size | 2147483647
+ repair_session_max_tree_depth | 20
+ repair_session_space_in_mb | 62
+ repaired_data_tracking_for_partition_reads_enabled | false
+ repaired_data_tracking_for_range_reads_enabled | false
+ report_unconfirmed_repaired_data_mismatches | false
+ request_timeout_in_ms | 10000
+ role_manager | CassandraRoleManager
+ rpc_address | localhost
+ rpc_interface | null
+ saved_caches_directory | /home/ec2-user/cassandra/data/saved_caches
+ seed_provider | org.apache.cassandra.locator.SimpleSeedProvider{seeds=127.0.0.1:7000}
+ server_encryption_options_algorithm | null
+ server_encryption_options_protocol | TLS
+ slow_query_log_timeout_in_ms | 500
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ transparent_data_encryption_options_chunk_length_kb | 64
+ transparent_data_encryption_options_cipher | AES/CBC/PKCS5Padding
+ transparent_data_encryption_options_enabled | false
+ transparent_data_encryption_options_iv_length |
+ 16
+ (224 rows)
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+As no sstable tasks are running initially the ``system_views.sstable_tasks`` table lists 0 rows.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+---------+------+----------+-------+------
+
+ (0 rows)
+
Review comment:
maybe add entry about table metrics post https://issues.apache.org/jira/browse/CASSANDRA-15194 from with examples like:
```
cqlsh> SELECT * FROM disk_usage WHERE mebibytes > 1 ALLOW FILTERING;
keyspace_name | table_name | mebibytes
---------------+------------+-----------
keyspace1 | standard1 | 288
tlp_stress | keyvalue | 3211
cqlsh> SELECT * FROM local_read_latency WHERE per_second > 1 ALLOW FILTERING;
keyspace_name | table_name | p50th_ms | p99th_ms | count | max_ms | per_second
---------------+------------+----------+----------+----------+---------+------------
tlp_stress | keyvalue | 0.043 | 0.152 | 49785158 | 186.563 | 11418.356
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379492400
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
Review comment:
remove `however`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] Deepak-Vohra commented on a change in pull request
#402: New/virtualtables
Posted by GitBox <gi...@apache.org>.
Deepak-Vohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r382865745
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,337 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose metrics through CQL
+- Expose YAML configuration information
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only, but it is likely to change
+- Virtual tables are not replicated
+- Virtual tables are local only and non distributed
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use ``LocalPartitioner``. Since a virtual table is not replicated the partitioner sorts in order of partition keys instead of by their hash.
+- Making advanced queries with ``ALLOW FILTERING`` and aggregation functions may be used with virtual tables even though in normal tables we dont recommend it
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces:
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables.
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations initially though some of these could change such as:
+
+- Cannot alter or drop virtual keyspaces or tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support ``DELETE`` statements
+- Cannot ``CREATE TRIGGER`` against a virtual table
+- Conditional ``BATCH`` statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional ``BATCH`` statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces; but may run aggregate functions on select
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ coordinator_scans clients tombstones_scanned internode_inbound
+ disk_usage sstable_tasks live_scanned caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in ``system_views`` virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name:
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+Some examples of how ``clients`` can be used are:
+
+- To find applications using old incompatible versions of drivers before upgrading and with ``nodetool enableoldprotocolversions`` and ``nodetool disableoldprotocolversions`` during upgrades.
+- To identify clients sending too many requests.
+- To find if SSL is enabled during the migration to and from ssl.
+
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table:
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details:
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings`` table is rather useful and lists all the current configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the virtual table presently:
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_enabled | false
+ auto_snapshot | true
+ automatic_sstable_upgrade | false
+ cluster_name | Test Cluster
+ enable_transient_replication | false
+ hinted_handoff_enabled | true
+ hints_directory | /home/ec2-user/cassandra/data/hints
+ incremental_backups | false
+ initial_token | null
+ ...
+ ...
+ ...
+ rpc_address | localhost
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ (224 rows)
+
+
+The ``settings`` table can be really useful if yaml file has been changed since startup and dont know running configuration, or to find if they have been modified via jmx/nodetool or virtual tables.
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details:
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details:
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+The ``sstable_tasks`` could be used to get information about running tasks. It lists following columns:
+
+::
+
+ cqlsh:system_views> SELECT * FROM sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+--------------------------------------+------------+----------+----------+-------
+ basic | wide2 | c3909740-cdf7-11e9-a8ed-0f03de2d9ae1 | compaction | 60418761 | 70882110 | bytes
+ basic | wide2 | c7556770-cdf7-11e9-a8ed-0f03de2d9ae1 | compaction | 2995623 | 40314679 | bytes
+
+
+As another example, a query ``SELECT (total-progress) FROM sstable_tasks;`` gives the remaining time for a task.
Review comment:
I have made the query valid.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379569285
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
Review comment:
Modified.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] Deepak-Vohra commented on a change in pull request
#402: New/virtualtables
Posted by GitBox <gi...@apache.org>.
Deepak-Vohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r380876617
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,329 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose metrics through CQL
+- Expose YAML configuration information
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only, but it is likely to change
+- Virtual tables are not replicated
+- Virtual tables are local only and non distributed
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use ``LocalPartitioner``. Since a virtual table is not replicated the partitioner sorts in order of partition keys instead of by their hash.
+- Making advanced queries with ``ALLOW FILTERING`` and aggregation functions may be used with virtual tables even though in normal tables we dont recommend it
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables.
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations initially though some of these could change such as:
+
+- Cannot alter or drop virtual keyspaces or tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces; but may run aggregate functions on select
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ coordinator_scans clients tombstones_scanned internode_inbound
+ disk_usage sstable_tasks live_scanned caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in ``system_views`` virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+Some examples of how ``clients`` can be used are:
+
+- To find applications using old incompatible versions of drivers before upgrading and with ``nodetool enableoldprotocolversions`` and ``nodetool disableoldprotocolversions`` during upgrades.
+- To identify clients sending too many requests.
+- To find if SSL is enabled during the migration to and from ssl.
+
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the current configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the virtual table presently.
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_enabled | false
+ auto_snapshot | true
+ automatic_sstable_upgrade | false
+ cluster_name | Test Cluster
+ enable_transient_replication | false
+ hinted_handoff_enabled | true
+ hints_directory | /home/ec2-user/cassandra/data/hints
+ incremental_backups | false
+ initial_token | null
+ ...
+ ...
+ ...
+ rpc_address | localhost
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ (224 rows)
+
+
+The ``settings`` table can be really useful if yaml file has been changed since startup and dont know running configuration, or to find if they have been modified via jmx/nodetool or virtual tables.
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+The ``sstable_tasks`` could be used to get information about running tasks. It lists following columns.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+---------+------+----------+-------+------
+
+
+As an example a query of ``total-progress`` gives the remaining time for a task.
+
+Other Virtual Tables
+********************
+
+Some examples of using other virtual tables are as follows.
+
+::
Review comment:
Added. Please double check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379571071
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the
+virtual table presently. A total of 224 settings get listed presently.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -----------------------------------------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_audit_logs_dir |
+ /home/ec2-user/cassandra/logs/audit/
+ audit_logging_options_enabled |
+ false
+ audit_logging_options_excluded_categories |
+ audit_logging_options_excluded_keyspaces |
+ system,system_schema,system_virtual_schema
+ audit_logging_options_excluded_users |
+ audit_logging_options_included_categories |
+ audit_logging_options_included_keyspaces |
+ audit_logging_options_included_users |
+ audit_logging_options_logger |
+ BinAuditLogger
+ authenticator |
+ AllowAllAuthenticator
+ authorizer |
+ AllowAllAuthorizer
+ auto_bootstrap |
+ true
+ auto_snapshot |
+ true
+ automatic_sstable_upgrade |
+ false
+ back_pressure_enabled |
+ false
+ back_pressure_strategy |
+ org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}
+ broadcast_address |
+ null
+ broadcast_rpc_address |
+ null
+ cluster_name |
+ Test Cluster
+ column_index_cache_size_in_kb |
+ 2
+ column_index_size_in_kb |
+ 64
+ commit_failure_policy |
+ stop
+ commitlog_compression |
+ null
+ commitlog_directory |
+ /home/ec2-user/cassandra/data/commitlog
+ commitlog_max_compression_buffers_in_pool |
+ 3
+ commitlog_periodic_queue_size |
+ -1
+ commitlog_segment_size_in_mb |
+ 32
+ commitlog_sync |
+ periodic
+ commitlog_sync_batch_window_in_ms |
+ NaN
+ commitlog_sync_group_window_in_ms |
+ NaN
+ commitlog_sync_period_in_ms |
+ 10000
+ commitlog_total_space_in_mb |
+ 2556
+ compaction_large_partition_warning_threshold_mb |
+ 100
+ compaction_throughput_mb_per_sec |
+ 16
+ concurrent_compactors |
+ 2
+ concurrent_counter_writes |
+ 32
+ concurrent_materialized_view_builders |
+ 1
+ concurrent_materialized_view_writes |
+ 32
+ concurrent_reads |
+ 32
+ concurrent_replicates |
+ null
+ concurrent_validations |
+ 2147483647
+ concurrent_writes |
+ 32
+ credentials_update_interval_in_ms |
+ -1
+ credentials_validity_in_ms |
+ 2000
+ cross_node_timeout |
+ false
+ data_file_directories |
+ [/home/ec2-user/cassandra/data/data]
+ diagnostic_events_enabled |
+ false
+ disk_access_mode |
+ mmap
+ disk_failure_policy |
+ stop
+ enable_materialized_views |
+ false
+ enable_sasi_indexes |
+ false
+ enable_scripted_user_defined_functions |
+ false
+ enable_transient_replication |
+ false
+ enable_user_defined_functions |
+ false
+ enable_user_defined_functions_threads |
+ true
+ endpoint_snitch |
+ SimpleSnitch
+ file_cache_round_up |
+ false
+ file_cache_size_in_mb |
+ 251
+ full_query_logging_options |
+ FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true,
+ max_queue_weight=268435456, max_log_size=17179869184}
+ gc_log_threshold_in_ms |
+ 200
+ gc_warn_threshold_in_ms |
+ 1000
+ hinted_handoff_disabled_datacenters |
+ []
+ hinted_handoff_enabled |
+ true
+ hinted_handoff_throttle_in_kb |
+ 1024
+ hints_compression |
+ null
+ hints_directory |
+ /home/ec2-user/cassandra/data/hints
+ hints_flush_period_in_ms |
+ 10000
+ ideal_consistency_level |
+ null
+ incremental_backups |
+ false
+ initial_token |
+ null
+ inter_dc_stream_throughput_outbound_megabits_per_sec |
+ 200
+ inter_dc_tcp_nodelay |
+ false
+ internode_application_receive_queue_capacity_in_bytes |
+ 4194304
+ internode_application_receive_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+ internode_application_receive_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_application_send_queue_capacity_in_bytes |
+ 4194304
+ internode_application_send_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+
+
+ internode_application_send_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_authenticator |
+ null
+ internode_compression |
+ dc
+ listen_address |
+ localhost
+ listen_interface |
+ null
+ listen_on_broadcast_address |
+ false
+ max_concurrent_automatic_sstable_upgrades |
+ 1
+ max_hint_window_in_ms |
+ 10800000
+ memtable_allocation_type |
+ heap_buffers
+ min_free_space_per_drive_in_mb |
+ 50
+ native_transport_allow_older_protocols |
+ true
+ native_transport_flush_in_batches_legacy |
+ false
+ native_transport_frame_block_size_in_kb | 32
+ native_transport_idle_timeout_in_ms | 0
+ native_transport_max_concurrent_connections | -1
+ native_transport_max_concurrent_connections_per_ip | -1
+ native_transport_max_concurrent_requests_in_bytes | 105277030
+ native_transport_max_concurrent_requests_in_bytes_per_ip | 26319257
+ native_transport_max_frame_size_in_mb | 256
+ native_transport_max_threads | 128
+ native_transport_port | 9042
+ native_transport_port_ssl | null
+ network_authorizer | AllowAllNetworkAuthorizer
+ num_tokens | 256
+ partitioner | org.apache.cassandra.dht.Murmur3Partitioner
+ prepared_statements_cache_size_mb | null
+ range_request_timeout_in_ms | 10000
+ read_request_timeout_in_ms | 5000
+ repair_command_pool_full_strategy | queue
+ repair_command_pool_size | 2147483647
+ repair_session_max_tree_depth | 20
+ repair_session_space_in_mb | 62
+ repaired_data_tracking_for_partition_reads_enabled | false
+ repaired_data_tracking_for_range_reads_enabled | false
+ report_unconfirmed_repaired_data_mismatches | false
+ request_timeout_in_ms | 10000
+ role_manager | CassandraRoleManager
+ rpc_address | localhost
+ rpc_interface | null
+ saved_caches_directory | /home/ec2-user/cassandra/data/saved_caches
+ seed_provider | org.apache.cassandra.locator.SimpleSeedProvider{seeds=127.0.0.1:7000}
+ server_encryption_options_algorithm | null
+ server_encryption_options_protocol | TLS
+ slow_query_log_timeout_in_ms | 500
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ transparent_data_encryption_options_chunk_length_kb | 64
+ transparent_data_encryption_options_cipher | AES/CBC/PKCS5Padding
+ transparent_data_encryption_options_enabled | false
+ transparent_data_encryption_options_iv_length |
+ 16
+ (224 rows)
Review comment:
Removed several listing settings. Added `...`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379575397
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
Review comment:
Added
- Cannot create aggregates in virtual keyspaces; but may run aggregate functions on select
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r380855759
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,329 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose metrics through CQL
+- Expose YAML configuration information
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only, but it is likely to change
+- Virtual tables are not replicated
+- Virtual tables are local only and non distributed
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use ``LocalPartitioner``. Since a virtual table is not replicated the partitioner sorts in order of partition keys instead of by their hash.
+- Making advanced queries with ``ALLOW FILTERING`` and aggregation functions may be used with virtual tables even though in normal tables we dont recommend it
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables.
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations initially though some of these could change such as:
+
+- Cannot alter or drop virtual keyspaces or tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces; but may run aggregate functions on select
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ coordinator_scans clients tombstones_scanned internode_inbound
+ disk_usage sstable_tasks live_scanned caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in ``system_views`` virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+Some examples of how ``clients`` can be used are:
+
+- To find applications using old incompatible versions of drivers before upgrading and with ``nodetool enableoldprotocolversions`` and ``nodetool disableoldprotocolversions`` during upgrades.
+- To identify clients sending too many requests.
+- To find if SSL is enabled during the migration to and from ssl.
+
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the current configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the virtual table presently.
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_enabled | false
+ auto_snapshot | true
+ automatic_sstable_upgrade | false
+ cluster_name | Test Cluster
+ enable_transient_replication | false
+ hinted_handoff_enabled | true
+ hints_directory | /home/ec2-user/cassandra/data/hints
+ incremental_backups | false
+ initial_token | null
+ ...
+ ...
+ ...
+ rpc_address | localhost
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ (224 rows)
+
+
+The ``settings`` table can be really useful if yaml file has been changed since startup and dont know running configuration, or to find if they have been modified via jmx/nodetool or virtual tables.
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+The ``sstable_tasks`` could be used to get information about running tasks. It lists following columns.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+---------+------+----------+-------+------
+
+
+As an example a query of ``total-progress`` gives the remaining time for a task.
Review comment:
whats total-progress ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379492996
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
Review comment:
I don't think this sentence is necessary
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379571959
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the
+virtual table presently. A total of 224 settings get listed presently.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -----------------------------------------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_audit_logs_dir |
+ /home/ec2-user/cassandra/logs/audit/
+ audit_logging_options_enabled |
+ false
+ audit_logging_options_excluded_categories |
+ audit_logging_options_excluded_keyspaces |
+ system,system_schema,system_virtual_schema
+ audit_logging_options_excluded_users |
+ audit_logging_options_included_categories |
+ audit_logging_options_included_keyspaces |
+ audit_logging_options_included_users |
+ audit_logging_options_logger |
+ BinAuditLogger
+ authenticator |
+ AllowAllAuthenticator
+ authorizer |
+ AllowAllAuthorizer
+ auto_bootstrap |
+ true
+ auto_snapshot |
+ true
+ automatic_sstable_upgrade |
+ false
+ back_pressure_enabled |
+ false
+ back_pressure_strategy |
+ org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}
+ broadcast_address |
+ null
+ broadcast_rpc_address |
+ null
+ cluster_name |
+ Test Cluster
+ column_index_cache_size_in_kb |
+ 2
+ column_index_size_in_kb |
+ 64
+ commit_failure_policy |
+ stop
+ commitlog_compression |
+ null
+ commitlog_directory |
+ /home/ec2-user/cassandra/data/commitlog
+ commitlog_max_compression_buffers_in_pool |
+ 3
+ commitlog_periodic_queue_size |
+ -1
+ commitlog_segment_size_in_mb |
+ 32
+ commitlog_sync |
+ periodic
+ commitlog_sync_batch_window_in_ms |
+ NaN
+ commitlog_sync_group_window_in_ms |
+ NaN
+ commitlog_sync_period_in_ms |
+ 10000
+ commitlog_total_space_in_mb |
+ 2556
+ compaction_large_partition_warning_threshold_mb |
+ 100
+ compaction_throughput_mb_per_sec |
+ 16
+ concurrent_compactors |
+ 2
+ concurrent_counter_writes |
+ 32
+ concurrent_materialized_view_builders |
+ 1
+ concurrent_materialized_view_writes |
+ 32
+ concurrent_reads |
+ 32
+ concurrent_replicates |
+ null
+ concurrent_validations |
+ 2147483647
+ concurrent_writes |
+ 32
+ credentials_update_interval_in_ms |
+ -1
+ credentials_validity_in_ms |
+ 2000
+ cross_node_timeout |
+ false
+ data_file_directories |
+ [/home/ec2-user/cassandra/data/data]
+ diagnostic_events_enabled |
+ false
+ disk_access_mode |
+ mmap
+ disk_failure_policy |
+ stop
+ enable_materialized_views |
+ false
+ enable_sasi_indexes |
+ false
+ enable_scripted_user_defined_functions |
+ false
+ enable_transient_replication |
+ false
+ enable_user_defined_functions |
+ false
+ enable_user_defined_functions_threads |
+ true
+ endpoint_snitch |
+ SimpleSnitch
+ file_cache_round_up |
+ false
+ file_cache_size_in_mb |
+ 251
+ full_query_logging_options |
+ FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true,
+ max_queue_weight=268435456, max_log_size=17179869184}
+ gc_log_threshold_in_ms |
+ 200
+ gc_warn_threshold_in_ms |
+ 1000
+ hinted_handoff_disabled_datacenters |
+ []
+ hinted_handoff_enabled |
+ true
+ hinted_handoff_throttle_in_kb |
+ 1024
+ hints_compression |
+ null
+ hints_directory |
+ /home/ec2-user/cassandra/data/hints
+ hints_flush_period_in_ms |
+ 10000
+ ideal_consistency_level |
+ null
+ incremental_backups |
+ false
+ initial_token |
+ null
+ inter_dc_stream_throughput_outbound_megabits_per_sec |
+ 200
+ inter_dc_tcp_nodelay |
+ false
+ internode_application_receive_queue_capacity_in_bytes |
+ 4194304
+ internode_application_receive_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+ internode_application_receive_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_application_send_queue_capacity_in_bytes |
+ 4194304
+ internode_application_send_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+
+
+ internode_application_send_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_authenticator |
+ null
+ internode_compression |
+ dc
+ listen_address |
+ localhost
+ listen_interface |
+ null
+ listen_on_broadcast_address |
+ false
+ max_concurrent_automatic_sstable_upgrades |
+ 1
+ max_hint_window_in_ms |
+ 10800000
+ memtable_allocation_type |
+ heap_buffers
+ min_free_space_per_drive_in_mb |
+ 50
+ native_transport_allow_older_protocols |
+ true
+ native_transport_flush_in_batches_legacy |
+ false
+ native_transport_frame_block_size_in_kb | 32
+ native_transport_idle_timeout_in_ms | 0
+ native_transport_max_concurrent_connections | -1
+ native_transport_max_concurrent_connections_per_ip | -1
+ native_transport_max_concurrent_requests_in_bytes | 105277030
+ native_transport_max_concurrent_requests_in_bytes_per_ip | 26319257
+ native_transport_max_frame_size_in_mb | 256
+ native_transport_max_threads | 128
+ native_transport_port | 9042
+ native_transport_port_ssl | null
+ network_authorizer | AllowAllNetworkAuthorizer
+ num_tokens | 256
+ partitioner | org.apache.cassandra.dht.Murmur3Partitioner
+ prepared_statements_cache_size_mb | null
+ range_request_timeout_in_ms | 10000
+ read_request_timeout_in_ms | 5000
+ repair_command_pool_full_strategy | queue
+ repair_command_pool_size | 2147483647
+ repair_session_max_tree_depth | 20
+ repair_session_space_in_mb | 62
+ repaired_data_tracking_for_partition_reads_enabled | false
+ repaired_data_tracking_for_range_reads_enabled | false
+ report_unconfirmed_repaired_data_mismatches | false
+ request_timeout_in_ms | 10000
+ role_manager | CassandraRoleManager
+ rpc_address | localhost
+ rpc_interface | null
+ saved_caches_directory | /home/ec2-user/cassandra/data/saved_caches
+ seed_provider | org.apache.cassandra.locator.SimpleSeedProvider{seeds=127.0.0.1:7000}
+ server_encryption_options_algorithm | null
+ server_encryption_options_protocol | TLS
+ slow_query_log_timeout_in_ms | 500
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ transparent_data_encryption_options_chunk_length_kb | 64
+ transparent_data_encryption_options_cipher | AES/CBC/PKCS5Padding
+ transparent_data_encryption_options_enabled | false
+ transparent_data_encryption_options_iv_length |
+ 16
+ (224 rows)
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+As no sstable tasks are running initially the ``system_views.sstable_tasks`` table lists 0 rows.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+---------+------+----------+-------+------
+
+ (0 rows)
+
+The system_virtual_schema keyspace
Review comment:
Removed most of the section and all sub-sections
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379600591
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the
+virtual table presently. A total of 224 settings get listed presently.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -----------------------------------------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_audit_logs_dir |
+ /home/ec2-user/cassandra/logs/audit/
+ audit_logging_options_enabled |
+ false
+ audit_logging_options_excluded_categories |
+ audit_logging_options_excluded_keyspaces |
+ system,system_schema,system_virtual_schema
+ audit_logging_options_excluded_users |
+ audit_logging_options_included_categories |
+ audit_logging_options_included_keyspaces |
+ audit_logging_options_included_users |
+ audit_logging_options_logger |
+ BinAuditLogger
+ authenticator |
+ AllowAllAuthenticator
+ authorizer |
+ AllowAllAuthorizer
+ auto_bootstrap |
+ true
+ auto_snapshot |
+ true
+ automatic_sstable_upgrade |
+ false
+ back_pressure_enabled |
+ false
+ back_pressure_strategy |
+ org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}
+ broadcast_address |
+ null
+ broadcast_rpc_address |
+ null
+ cluster_name |
+ Test Cluster
+ column_index_cache_size_in_kb |
+ 2
+ column_index_size_in_kb |
+ 64
+ commit_failure_policy |
+ stop
+ commitlog_compression |
+ null
+ commitlog_directory |
+ /home/ec2-user/cassandra/data/commitlog
+ commitlog_max_compression_buffers_in_pool |
+ 3
+ commitlog_periodic_queue_size |
+ -1
+ commitlog_segment_size_in_mb |
+ 32
+ commitlog_sync |
+ periodic
+ commitlog_sync_batch_window_in_ms |
+ NaN
+ commitlog_sync_group_window_in_ms |
+ NaN
+ commitlog_sync_period_in_ms |
+ 10000
+ commitlog_total_space_in_mb |
+ 2556
+ compaction_large_partition_warning_threshold_mb |
+ 100
+ compaction_throughput_mb_per_sec |
+ 16
+ concurrent_compactors |
+ 2
+ concurrent_counter_writes |
+ 32
+ concurrent_materialized_view_builders |
+ 1
+ concurrent_materialized_view_writes |
+ 32
+ concurrent_reads |
+ 32
+ concurrent_replicates |
+ null
+ concurrent_validations |
+ 2147483647
+ concurrent_writes |
+ 32
+ credentials_update_interval_in_ms |
+ -1
+ credentials_validity_in_ms |
+ 2000
+ cross_node_timeout |
+ false
+ data_file_directories |
+ [/home/ec2-user/cassandra/data/data]
+ diagnostic_events_enabled |
+ false
+ disk_access_mode |
+ mmap
+ disk_failure_policy |
+ stop
+ enable_materialized_views |
+ false
+ enable_sasi_indexes |
+ false
+ enable_scripted_user_defined_functions |
+ false
+ enable_transient_replication |
+ false
+ enable_user_defined_functions |
+ false
+ enable_user_defined_functions_threads |
+ true
+ endpoint_snitch |
+ SimpleSnitch
+ file_cache_round_up |
+ false
+ file_cache_size_in_mb |
+ 251
+ full_query_logging_options |
+ FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true,
+ max_queue_weight=268435456, max_log_size=17179869184}
+ gc_log_threshold_in_ms |
+ 200
+ gc_warn_threshold_in_ms |
+ 1000
+ hinted_handoff_disabled_datacenters |
+ []
+ hinted_handoff_enabled |
+ true
+ hinted_handoff_throttle_in_kb |
+ 1024
+ hints_compression |
+ null
+ hints_directory |
+ /home/ec2-user/cassandra/data/hints
+ hints_flush_period_in_ms |
+ 10000
+ ideal_consistency_level |
+ null
+ incremental_backups |
+ false
+ initial_token |
+ null
+ inter_dc_stream_throughput_outbound_megabits_per_sec |
+ 200
+ inter_dc_tcp_nodelay |
+ false
+ internode_application_receive_queue_capacity_in_bytes |
+ 4194304
+ internode_application_receive_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+ internode_application_receive_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_application_send_queue_capacity_in_bytes |
+ 4194304
+ internode_application_send_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+
+
+ internode_application_send_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_authenticator |
+ null
+ internode_compression |
+ dc
+ listen_address |
+ localhost
+ listen_interface |
+ null
+ listen_on_broadcast_address |
+ false
+ max_concurrent_automatic_sstable_upgrades |
+ 1
+ max_hint_window_in_ms |
+ 10800000
+ memtable_allocation_type |
+ heap_buffers
+ min_free_space_per_drive_in_mb |
+ 50
+ native_transport_allow_older_protocols |
+ true
+ native_transport_flush_in_batches_legacy |
+ false
+ native_transport_frame_block_size_in_kb | 32
+ native_transport_idle_timeout_in_ms | 0
+ native_transport_max_concurrent_connections | -1
+ native_transport_max_concurrent_connections_per_ip | -1
+ native_transport_max_concurrent_requests_in_bytes | 105277030
+ native_transport_max_concurrent_requests_in_bytes_per_ip | 26319257
+ native_transport_max_frame_size_in_mb | 256
+ native_transport_max_threads | 128
+ native_transport_port | 9042
+ native_transport_port_ssl | null
+ network_authorizer | AllowAllNetworkAuthorizer
+ num_tokens | 256
+ partitioner | org.apache.cassandra.dht.Murmur3Partitioner
+ prepared_statements_cache_size_mb | null
+ range_request_timeout_in_ms | 10000
+ read_request_timeout_in_ms | 5000
+ repair_command_pool_full_strategy | queue
+ repair_command_pool_size | 2147483647
+ repair_session_max_tree_depth | 20
+ repair_session_space_in_mb | 62
+ repaired_data_tracking_for_partition_reads_enabled | false
+ repaired_data_tracking_for_range_reads_enabled | false
+ report_unconfirmed_repaired_data_mismatches | false
+ request_timeout_in_ms | 10000
+ role_manager | CassandraRoleManager
+ rpc_address | localhost
+ rpc_interface | null
+ saved_caches_directory | /home/ec2-user/cassandra/data/saved_caches
+ seed_provider | org.apache.cassandra.locator.SimpleSeedProvider{seeds=127.0.0.1:7000}
+ server_encryption_options_algorithm | null
+ server_encryption_options_protocol | TLS
+ slow_query_log_timeout_in_ms | 500
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ transparent_data_encryption_options_chunk_length_kb | 64
+ transparent_data_encryption_options_cipher | AES/CBC/PKCS5Padding
+ transparent_data_encryption_options_enabled | false
+ transparent_data_encryption_options_iv_length |
+ 16
+ (224 rows)
Review comment:
Removed scroll. Only the basic settings are listed such as cluster name and storage port.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] Deepak-Vohra commented on a change in pull request
#402: New/virtualtables
Posted by GitBox <gi...@apache.org>.
Deepak-Vohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r380876449
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,329 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose metrics through CQL
+- Expose YAML configuration information
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only, but it is likely to change
+- Virtual tables are not replicated
+- Virtual tables are local only and non distributed
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use ``LocalPartitioner``. Since a virtual table is not replicated the partitioner sorts in order of partition keys instead of by their hash.
+- Making advanced queries with ``ALLOW FILTERING`` and aggregation functions may be used with virtual tables even though in normal tables we dont recommend it
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables.
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations initially though some of these could change such as:
+
+- Cannot alter or drop virtual keyspaces or tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces; but may run aggregate functions on select
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ coordinator_scans clients tombstones_scanned internode_inbound
+ disk_usage sstable_tasks live_scanned caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in ``system_views`` virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+Some examples of how ``clients`` can be used are:
+
+- To find applications using old incompatible versions of drivers before upgrading and with ``nodetool enableoldprotocolversions`` and ``nodetool disableoldprotocolversions`` during upgrades.
+- To identify clients sending too many requests.
+- To find if SSL is enabled during the migration to and from ssl.
+
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the current configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the virtual table presently.
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_enabled | false
+ auto_snapshot | true
+ automatic_sstable_upgrade | false
+ cluster_name | Test Cluster
+ enable_transient_replication | false
+ hinted_handoff_enabled | true
+ hints_directory | /home/ec2-user/cassandra/data/hints
+ incremental_backups | false
+ initial_token | null
+ ...
+ ...
+ ...
+ rpc_address | localhost
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ (224 rows)
+
+
+The ``settings`` table can be really useful if yaml file has been changed since startup and dont know running configuration, or to find if they have been modified via jmx/nodetool or virtual tables.
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+The ``sstable_tasks`` could be used to get information about running tasks. It lists following columns.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+---------+------+----------+-------+------
+
+
+As an example a query of ``total-progress`` gives the remaining time for a task.
Review comment:
Added an example query for finding remaining time in a task.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379567935
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
Review comment:
Added examples.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r380858514
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,329 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose metrics through CQL
+- Expose YAML configuration information
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only, but it is likely to change
+- Virtual tables are not replicated
+- Virtual tables are local only and non distributed
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use ``LocalPartitioner``. Since a virtual table is not replicated the partitioner sorts in order of partition keys instead of by their hash.
+- Making advanced queries with ``ALLOW FILTERING`` and aggregation functions may be used with virtual tables even though in normal tables we dont recommend it
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables.
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations initially though some of these could change such as:
+
+- Cannot alter or drop virtual keyspaces or tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces; but may run aggregate functions on select
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ coordinator_scans clients tombstones_scanned internode_inbound
+ disk_usage sstable_tasks live_scanned caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in ``system_views`` virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+Some examples of how ``clients`` can be used are:
+
+- To find applications using old incompatible versions of drivers before upgrading and with ``nodetool enableoldprotocolversions`` and ``nodetool disableoldprotocolversions`` during upgrades.
+- To identify clients sending too many requests.
+- To find if SSL is enabled during the migration to and from ssl.
+
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the current configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the virtual table presently.
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_enabled | false
+ auto_snapshot | true
+ automatic_sstable_upgrade | false
+ cluster_name | Test Cluster
+ enable_transient_replication | false
+ hinted_handoff_enabled | true
+ hints_directory | /home/ec2-user/cassandra/data/hints
+ incremental_backups | false
+ initial_token | null
+ ...
+ ...
+ ...
+ rpc_address | localhost
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ (224 rows)
+
+
+The ``settings`` table can be really useful if yaml file has been changed since startup and dont know running configuration, or to find if they have been modified via jmx/nodetool or virtual tables.
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+The ``sstable_tasks`` could be used to get information about running tasks. It lists following columns.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+---------+------+----------+-------+------
+
+
+As an example a query of ``total-progress`` gives the remaining time for a task.
+
+Other Virtual Tables
+********************
+
+Some examples of using other virtual tables are as follows.
+
+::
Review comment:
can add comment before each of these like "Find tables with most disk usage" and "find most queried tables"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379491666
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
Review comment:
kinda covered with unable to create functions and maybe mis understood as not able to run aggregate functions on select which it does allow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379568915
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
Review comment:
Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379490158
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
Review comment:
this isnt true, just the existing ones dont support it. Each table can choose to support different features really in terms of what kind of SELECT and INSERT/UPDATE/DELETE they handle (ie expiring cells and TTL isnt explicitly forbidden, just nothing _currently_ existing supports it since we going out initially with read only metrics essentially
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] Deepak-Vohra commented on a change in pull request
#402: New/virtualtables
Posted by GitBox <gi...@apache.org>.
Deepak-Vohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r382865852
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,329 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose metrics through CQL
+- Expose YAML configuration information
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only, but it is likely to change
+- Virtual tables are not replicated
+- Virtual tables are local only and non distributed
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use ``LocalPartitioner``. Since a virtual table is not replicated the partitioner sorts in order of partition keys instead of by their hash.
+- Making advanced queries with ``ALLOW FILTERING`` and aggregation functions may be used with virtual tables even though in normal tables we dont recommend it
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables.
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations initially though some of these could change such as:
+
+- Cannot alter or drop virtual keyspaces or tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces; but may run aggregate functions on select
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ coordinator_scans clients tombstones_scanned internode_inbound
+ disk_usage sstable_tasks live_scanned caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in ``system_views`` virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+Some examples of how ``clients`` can be used are:
+
+- To find applications using old incompatible versions of drivers before upgrading and with ``nodetool enableoldprotocolversions`` and ``nodetool disableoldprotocolversions`` during upgrades.
+- To identify clients sending too many requests.
+- To find if SSL is enabled during the migration to and from ssl.
+
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the current configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the virtual table presently.
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_enabled | false
+ auto_snapshot | true
+ automatic_sstable_upgrade | false
+ cluster_name | Test Cluster
+ enable_transient_replication | false
+ hinted_handoff_enabled | true
+ hints_directory | /home/ec2-user/cassandra/data/hints
+ incremental_backups | false
+ initial_token | null
+ ...
+ ...
+ ...
+ rpc_address | localhost
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ (224 rows)
+
+
+The ``settings`` table can be really useful if yaml file has been changed since startup and dont know running configuration, or to find if they have been modified via jmx/nodetool or virtual tables.
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+The ``sstable_tasks`` could be used to get information about running tasks. It lists following columns.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+---------+------+----------+-------+------
+
+
+As an example a query of ``total-progress`` gives the remaining time for a task.
+
+Other Virtual Tables
+********************
+
+Some examples of using other virtual tables are as follows.
+
+::
Review comment:
modified statement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379561053
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
Review comment:
Removed JMX
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379561239
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
Review comment:
Removed mbeans/jmx
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379568765
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the
+virtual table presently. A total of 224 settings get listed presently.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -----------------------------------------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_audit_logs_dir |
+ /home/ec2-user/cassandra/logs/audit/
+ audit_logging_options_enabled |
+ false
+ audit_logging_options_excluded_categories |
+ audit_logging_options_excluded_keyspaces |
+ system,system_schema,system_virtual_schema
+ audit_logging_options_excluded_users |
+ audit_logging_options_included_categories |
+ audit_logging_options_included_keyspaces |
+ audit_logging_options_included_users |
+ audit_logging_options_logger |
+ BinAuditLogger
+ authenticator |
+ AllowAllAuthenticator
+ authorizer |
+ AllowAllAuthorizer
+ auto_bootstrap |
+ true
+ auto_snapshot |
+ true
+ automatic_sstable_upgrade |
+ false
+ back_pressure_enabled |
+ false
+ back_pressure_strategy |
+ org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}
+ broadcast_address |
+ null
+ broadcast_rpc_address |
+ null
+ cluster_name |
+ Test Cluster
+ column_index_cache_size_in_kb |
+ 2
+ column_index_size_in_kb |
+ 64
+ commit_failure_policy |
+ stop
+ commitlog_compression |
+ null
+ commitlog_directory |
+ /home/ec2-user/cassandra/data/commitlog
+ commitlog_max_compression_buffers_in_pool |
+ 3
+ commitlog_periodic_queue_size |
+ -1
+ commitlog_segment_size_in_mb |
+ 32
+ commitlog_sync |
+ periodic
+ commitlog_sync_batch_window_in_ms |
+ NaN
+ commitlog_sync_group_window_in_ms |
+ NaN
+ commitlog_sync_period_in_ms |
+ 10000
+ commitlog_total_space_in_mb |
+ 2556
+ compaction_large_partition_warning_threshold_mb |
+ 100
+ compaction_throughput_mb_per_sec |
+ 16
+ concurrent_compactors |
+ 2
+ concurrent_counter_writes |
+ 32
+ concurrent_materialized_view_builders |
+ 1
+ concurrent_materialized_view_writes |
+ 32
+ concurrent_reads |
+ 32
+ concurrent_replicates |
+ null
+ concurrent_validations |
+ 2147483647
+ concurrent_writes |
+ 32
+ credentials_update_interval_in_ms |
+ -1
+ credentials_validity_in_ms |
+ 2000
+ cross_node_timeout |
+ false
+ data_file_directories |
+ [/home/ec2-user/cassandra/data/data]
+ diagnostic_events_enabled |
+ false
+ disk_access_mode |
+ mmap
+ disk_failure_policy |
+ stop
+ enable_materialized_views |
+ false
+ enable_sasi_indexes |
+ false
+ enable_scripted_user_defined_functions |
+ false
+ enable_transient_replication |
+ false
+ enable_user_defined_functions |
+ false
+ enable_user_defined_functions_threads |
+ true
+ endpoint_snitch |
+ SimpleSnitch
+ file_cache_round_up |
+ false
+ file_cache_size_in_mb |
+ 251
+ full_query_logging_options |
+ FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true,
+ max_queue_weight=268435456, max_log_size=17179869184}
+ gc_log_threshold_in_ms |
+ 200
+ gc_warn_threshold_in_ms |
+ 1000
+ hinted_handoff_disabled_datacenters |
+ []
+ hinted_handoff_enabled |
+ true
+ hinted_handoff_throttle_in_kb |
+ 1024
+ hints_compression |
+ null
+ hints_directory |
+ /home/ec2-user/cassandra/data/hints
+ hints_flush_period_in_ms |
+ 10000
+ ideal_consistency_level |
+ null
+ incremental_backups |
+ false
+ initial_token |
+ null
+ inter_dc_stream_throughput_outbound_megabits_per_sec |
+ 200
+ inter_dc_tcp_nodelay |
+ false
+ internode_application_receive_queue_capacity_in_bytes |
+ 4194304
+ internode_application_receive_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+ internode_application_receive_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_application_send_queue_capacity_in_bytes |
+ 4194304
+ internode_application_send_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+
+
+ internode_application_send_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_authenticator |
+ null
+ internode_compression |
+ dc
+ listen_address |
+ localhost
+ listen_interface |
+ null
+ listen_on_broadcast_address |
+ false
+ max_concurrent_automatic_sstable_upgrades |
+ 1
+ max_hint_window_in_ms |
+ 10800000
+ memtable_allocation_type |
+ heap_buffers
+ min_free_space_per_drive_in_mb |
+ 50
+ native_transport_allow_older_protocols |
+ true
+ native_transport_flush_in_batches_legacy |
+ false
+ native_transport_frame_block_size_in_kb | 32
+ native_transport_idle_timeout_in_ms | 0
+ native_transport_max_concurrent_connections | -1
+ native_transport_max_concurrent_connections_per_ip | -1
+ native_transport_max_concurrent_requests_in_bytes | 105277030
+ native_transport_max_concurrent_requests_in_bytes_per_ip | 26319257
+ native_transport_max_frame_size_in_mb | 256
+ native_transport_max_threads | 128
+ native_transport_port | 9042
+ native_transport_port_ssl | null
+ network_authorizer | AllowAllNetworkAuthorizer
+ num_tokens | 256
+ partitioner | org.apache.cassandra.dht.Murmur3Partitioner
+ prepared_statements_cache_size_mb | null
+ range_request_timeout_in_ms | 10000
+ read_request_timeout_in_ms | 5000
+ repair_command_pool_full_strategy | queue
+ repair_command_pool_size | 2147483647
+ repair_session_max_tree_depth | 20
+ repair_session_space_in_mb | 62
+ repaired_data_tracking_for_partition_reads_enabled | false
+ repaired_data_tracking_for_range_reads_enabled | false
+ report_unconfirmed_repaired_data_mismatches | false
+ request_timeout_in_ms | 10000
+ role_manager | CassandraRoleManager
+ rpc_address | localhost
+ rpc_interface | null
+ saved_caches_directory | /home/ec2-user/cassandra/data/saved_caches
+ seed_provider | org.apache.cassandra.locator.SimpleSeedProvider{seeds=127.0.0.1:7000}
+ server_encryption_options_algorithm | null
+ server_encryption_options_protocol | TLS
+ slow_query_log_timeout_in_ms | 500
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ transparent_data_encryption_options_chunk_length_kb | 64
+ transparent_data_encryption_options_cipher | AES/CBC/PKCS5Padding
+ transparent_data_encryption_options_enabled | false
+ transparent_data_encryption_options_iv_length |
+ 16
+ (224 rows)
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+As no sstable tasks are running initially the ``system_views.sstable_tasks`` table lists 0 rows.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+---------+------+----------+-------+------
+
+ (0 rows)
+
Review comment:
Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379509776
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
Review comment:
Instead of just listing what these virtual tables are, I think it would be real valuable to list _why_ you would want to look at them. for example this can be used for finding applications using old incompatible versions of drivers before upgrading and with nodetool enableoldprotocolversions and nodetool
disableoldprotocolversions during upgrades. It can also be used to identify misbehaving clients sending too many requests or during the migration to and from ssl.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379581927
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the
+virtual table presently. A total of 224 settings get listed presently.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -----------------------------------------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_audit_logs_dir |
+ /home/ec2-user/cassandra/logs/audit/
+ audit_logging_options_enabled |
+ false
+ audit_logging_options_excluded_categories |
+ audit_logging_options_excluded_keyspaces |
+ system,system_schema,system_virtual_schema
+ audit_logging_options_excluded_users |
+ audit_logging_options_included_categories |
+ audit_logging_options_included_keyspaces |
+ audit_logging_options_included_users |
+ audit_logging_options_logger |
+ BinAuditLogger
+ authenticator |
+ AllowAllAuthenticator
+ authorizer |
+ AllowAllAuthorizer
+ auto_bootstrap |
+ true
+ auto_snapshot |
+ true
+ automatic_sstable_upgrade |
+ false
+ back_pressure_enabled |
+ false
+ back_pressure_strategy |
+ org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}
+ broadcast_address |
+ null
+ broadcast_rpc_address |
+ null
+ cluster_name |
+ Test Cluster
+ column_index_cache_size_in_kb |
+ 2
+ column_index_size_in_kb |
+ 64
+ commit_failure_policy |
+ stop
+ commitlog_compression |
+ null
+ commitlog_directory |
+ /home/ec2-user/cassandra/data/commitlog
+ commitlog_max_compression_buffers_in_pool |
+ 3
+ commitlog_periodic_queue_size |
+ -1
+ commitlog_segment_size_in_mb |
+ 32
+ commitlog_sync |
+ periodic
+ commitlog_sync_batch_window_in_ms |
+ NaN
+ commitlog_sync_group_window_in_ms |
+ NaN
+ commitlog_sync_period_in_ms |
+ 10000
+ commitlog_total_space_in_mb |
+ 2556
+ compaction_large_partition_warning_threshold_mb |
+ 100
+ compaction_throughput_mb_per_sec |
+ 16
+ concurrent_compactors |
+ 2
+ concurrent_counter_writes |
+ 32
+ concurrent_materialized_view_builders |
+ 1
+ concurrent_materialized_view_writes |
+ 32
+ concurrent_reads |
+ 32
+ concurrent_replicates |
+ null
+ concurrent_validations |
+ 2147483647
+ concurrent_writes |
+ 32
+ credentials_update_interval_in_ms |
+ -1
+ credentials_validity_in_ms |
+ 2000
+ cross_node_timeout |
+ false
+ data_file_directories |
+ [/home/ec2-user/cassandra/data/data]
+ diagnostic_events_enabled |
+ false
+ disk_access_mode |
+ mmap
+ disk_failure_policy |
+ stop
+ enable_materialized_views |
+ false
+ enable_sasi_indexes |
+ false
+ enable_scripted_user_defined_functions |
+ false
+ enable_transient_replication |
+ false
+ enable_user_defined_functions |
+ false
+ enable_user_defined_functions_threads |
+ true
+ endpoint_snitch |
+ SimpleSnitch
+ file_cache_round_up |
+ false
+ file_cache_size_in_mb |
+ 251
+ full_query_logging_options |
+ FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true,
+ max_queue_weight=268435456, max_log_size=17179869184}
+ gc_log_threshold_in_ms |
+ 200
+ gc_warn_threshold_in_ms |
+ 1000
+ hinted_handoff_disabled_datacenters |
+ []
+ hinted_handoff_enabled |
+ true
+ hinted_handoff_throttle_in_kb |
+ 1024
+ hints_compression |
+ null
+ hints_directory |
+ /home/ec2-user/cassandra/data/hints
+ hints_flush_period_in_ms |
+ 10000
+ ideal_consistency_level |
+ null
+ incremental_backups |
+ false
+ initial_token |
+ null
+ inter_dc_stream_throughput_outbound_megabits_per_sec |
+ 200
+ inter_dc_tcp_nodelay |
+ false
+ internode_application_receive_queue_capacity_in_bytes |
+ 4194304
+ internode_application_receive_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+ internode_application_receive_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_application_send_queue_capacity_in_bytes |
+ 4194304
+ internode_application_send_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+
+
+ internode_application_send_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_authenticator |
+ null
+ internode_compression |
+ dc
+ listen_address |
+ localhost
+ listen_interface |
+ null
+ listen_on_broadcast_address |
+ false
+ max_concurrent_automatic_sstable_upgrades |
+ 1
+ max_hint_window_in_ms |
+ 10800000
+ memtable_allocation_type |
+ heap_buffers
+ min_free_space_per_drive_in_mb |
+ 50
+ native_transport_allow_older_protocols |
+ true
+ native_transport_flush_in_batches_legacy |
+ false
+ native_transport_frame_block_size_in_kb | 32
+ native_transport_idle_timeout_in_ms | 0
+ native_transport_max_concurrent_connections | -1
+ native_transport_max_concurrent_connections_per_ip | -1
+ native_transport_max_concurrent_requests_in_bytes | 105277030
+ native_transport_max_concurrent_requests_in_bytes_per_ip | 26319257
+ native_transport_max_frame_size_in_mb | 256
+ native_transport_max_threads | 128
+ native_transport_port | 9042
+ native_transport_port_ssl | null
+ network_authorizer | AllowAllNetworkAuthorizer
+ num_tokens | 256
+ partitioner | org.apache.cassandra.dht.Murmur3Partitioner
+ prepared_statements_cache_size_mb | null
+ range_request_timeout_in_ms | 10000
+ read_request_timeout_in_ms | 5000
+ repair_command_pool_full_strategy | queue
+ repair_command_pool_size | 2147483647
+ repair_session_max_tree_depth | 20
+ repair_session_space_in_mb | 62
+ repaired_data_tracking_for_partition_reads_enabled | false
+ repaired_data_tracking_for_range_reads_enabled | false
+ report_unconfirmed_repaired_data_mismatches | false
+ request_timeout_in_ms | 10000
+ role_manager | CassandraRoleManager
+ rpc_address | localhost
+ rpc_interface | null
+ saved_caches_directory | /home/ec2-user/cassandra/data/saved_caches
+ seed_provider | org.apache.cassandra.locator.SimpleSeedProvider{seeds=127.0.0.1:7000}
+ server_encryption_options_algorithm | null
+ server_encryption_options_protocol | TLS
+ slow_query_log_timeout_in_ms | 500
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ transparent_data_encryption_options_chunk_length_kb | 64
+ transparent_data_encryption_options_cipher | AES/CBC/PKCS5Padding
+ transparent_data_encryption_options_enabled | false
+ transparent_data_encryption_options_iv_length |
+ 16
+ (224 rows)
Review comment:
i was thinking more along the lines of only listing like 10ish settings to give idea of output so its not this huge scroll. We dont really need an exhaustive list of all settings since that will be very hard to keep correct and is just a big data dump.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379568536
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
Review comment:
Added all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379570827
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the
+virtual table presently. A total of 224 settings get listed presently.
Review comment:
Removed how many settings.
Added:
- "all current configuration"
- This can be really useful if yaml file has been changed since startup and dont know running configuration or if concerned that they have been modified via jmx/nodetool or virtual tables.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379519201
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
Review comment:
Seems like some of those got cut off in the list above and shortened.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379517843
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
Review comment:
current list is
```
cqlsh> desc tables;
...
Keyspace system_views
---------------------
sstable_tasks
disk_usage
thread_pools
local_scan_latency
coordinator_scan_latency
max_partition_size
internode_inbound
coordinator_read_latency
caches
local_read_latency
rows_per_read
clients
local_write_latency
internode_outbound
coordinator_write_latency
tombstones_per_read
settings
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] rustyrazorblade closed pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
rustyrazorblade closed pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379486460
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
Review comment:
It does not expose JMX data, can just list expose metrics through CQL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379569393
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
Review comment:
removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379510795
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the
+virtual table presently. A total of 224 settings get listed presently.
Review comment:
This can be really useful if yaml file has been changed since startup and dont know running configuration or if concerned that they have been modified via jmx/nodetool or virtual tables.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379489690
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
Review comment:
Thats due to a bug in the driver that is fixed upstream and will soon be resolved and in trunk, DESCRIBE works on both virtual keyspaces and virtual tables - its just broken in trunk due to using older driver see https://issues.apache.org/jira/browse/CASSANDRA-14872
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379487493
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
Review comment:
Virtual tables are not read-only, although I think all the ones currently implemented are that will change very soon so best not to put that in docs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r382795469
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,337 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose metrics through CQL
+- Expose YAML configuration information
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only, but it is likely to change
+- Virtual tables are not replicated
+- Virtual tables are local only and non distributed
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use ``LocalPartitioner``. Since a virtual table is not replicated the partitioner sorts in order of partition keys instead of by their hash.
+- Making advanced queries with ``ALLOW FILTERING`` and aggregation functions may be used with virtual tables even though in normal tables we dont recommend it
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces:
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables.
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations initially though some of these could change such as:
+
+- Cannot alter or drop virtual keyspaces or tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support ``DELETE`` statements
+- Cannot ``CREATE TRIGGER`` against a virtual table
+- Conditional ``BATCH`` statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional ``BATCH`` statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces; but may run aggregate functions on select
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ coordinator_scans clients tombstones_scanned internode_inbound
+ disk_usage sstable_tasks live_scanned caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in ``system_views`` virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name:
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+Some examples of how ``clients`` can be used are:
+
+- To find applications using old incompatible versions of drivers before upgrading and with ``nodetool enableoldprotocolversions`` and ``nodetool disableoldprotocolversions`` during upgrades.
+- To identify clients sending too many requests.
+- To find if SSL is enabled during the migration to and from ssl.
+
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table:
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details:
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings`` table is rather useful and lists all the current configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the virtual table presently:
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_enabled | false
+ auto_snapshot | true
+ automatic_sstable_upgrade | false
+ cluster_name | Test Cluster
+ enable_transient_replication | false
+ hinted_handoff_enabled | true
+ hints_directory | /home/ec2-user/cassandra/data/hints
+ incremental_backups | false
+ initial_token | null
+ ...
+ ...
+ ...
+ rpc_address | localhost
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ (224 rows)
+
+
+The ``settings`` table can be really useful if yaml file has been changed since startup and dont know running configuration, or to find if they have been modified via jmx/nodetool or virtual tables.
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details:
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details:
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+The ``sstable_tasks`` could be used to get information about running tasks. It lists following columns:
+
+::
+
+ cqlsh:system_views> SELECT * FROM sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+--------------------------------------+------------+----------+----------+-------
+ basic | wide2 | c3909740-cdf7-11e9-a8ed-0f03de2d9ae1 | compaction | 60418761 | 70882110 | bytes
+ basic | wide2 | c7556770-cdf7-11e9-a8ed-0f03de2d9ae1 | compaction | 2995623 | 40314679 | bytes
+
+
+As another example, a query ``SELECT (total-progress) FROM sstable_tasks;`` gives the remaining time for a task.
Review comment:
thats not a valid query, remove the ()'s
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379495571
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the
+virtual table presently. A total of 224 settings get listed presently.
Review comment:
I wouldnt include number of settings since that can change constantly and will certainly be out of sync before 4.0 even.
May want to include that its "all current configuration". As configs can be changed at runtime (ie using nodetool setcompactionthroughput) which will be reflected here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379492022
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
Review comment:
I think you can put the above 4 in a single line like "cannot alter or drop virtual keyspace and tables"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379570237
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
Review comment:
Kept for current settings. Could be updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379566143
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
Review comment:
Removed statement about DESCRIBE.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379561875
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
Review comment:
Rather than list that is not currently supported, I have added:
- Virtual tables are read-only, but it is likely to change
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r380855386
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,329 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose metrics through CQL
+- Expose YAML configuration information
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only, but it is likely to change
+- Virtual tables are not replicated
+- Virtual tables are local only and non distributed
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use ``LocalPartitioner``. Since a virtual table is not replicated the partitioner sorts in order of partition keys instead of by their hash.
+- Making advanced queries with ``ALLOW FILTERING`` and aggregation functions may be used with virtual tables even though in normal tables we dont recommend it
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables.
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations initially though some of these could change such as:
+
+- Cannot alter or drop virtual keyspaces or tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces; but may run aggregate functions on select
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ coordinator_scans clients tombstones_scanned internode_inbound
+ disk_usage sstable_tasks live_scanned caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in ``system_views`` virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+Some examples of how ``clients`` can be used are:
+
+- To find applications using old incompatible versions of drivers before upgrading and with ``nodetool enableoldprotocolversions`` and ``nodetool disableoldprotocolversions`` during upgrades.
+- To identify clients sending too many requests.
+- To find if SSL is enabled during the migration to and from ssl.
+
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the current configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the virtual table presently.
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_enabled | false
+ auto_snapshot | true
+ automatic_sstable_upgrade | false
+ cluster_name | Test Cluster
+ enable_transient_replication | false
+ hinted_handoff_enabled | true
+ hints_directory | /home/ec2-user/cassandra/data/hints
+ incremental_backups | false
+ initial_token | null
+ ...
+ ...
+ ...
+ rpc_address | localhost
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ (224 rows)
+
+
+The ``settings`` table can be really useful if yaml file has been changed since startup and dont know running configuration, or to find if they have been modified via jmx/nodetool or virtual tables.
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+The ``sstable_tasks`` could be used to get information about running tasks. It lists following columns.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+---------+------+----------+-------+------
Review comment:
example with some content
```
cqlsh:system_views> SELECT * FROM sstable_tasks;
keyspace_name | table_name | task_id | kind | progress | total | unit
---------------+------------+--------------------------------------+------------+----------+----------+-------
basic | wide2 | c3909740-cdf7-11e9-a8ed-0f03de2d9ae1 | compaction | 60418761 | 70882110 | bytes
basic | wide2 | c7556770-cdf7-11e9-a8ed-0f03de2d9ae1 | compaction | 2995623 | 40314679 | bytes
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379513446
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
Review comment:
For LocalPartitioner I am not sure if people will necessarily know what that means so maybe spell out that since its not replicated (which you have above) the partitioner sorts in order of partition keys instead of by their hash.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r382796117
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,329 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose metrics through CQL
+- Expose YAML configuration information
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only, but it is likely to change
+- Virtual tables are not replicated
+- Virtual tables are local only and non distributed
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use ``LocalPartitioner``. Since a virtual table is not replicated the partitioner sorts in order of partition keys instead of by their hash.
+- Making advanced queries with ``ALLOW FILTERING`` and aggregation functions may be used with virtual tables even though in normal tables we dont recommend it
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables.
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations initially though some of these could change such as:
+
+- Cannot alter or drop virtual keyspaces or tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces; but may run aggregate functions on select
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ coordinator_scans clients tombstones_scanned internode_inbound
+ disk_usage sstable_tasks live_scanned caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in ``system_views`` virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+Some examples of how ``clients`` can be used are:
+
+- To find applications using old incompatible versions of drivers before upgrading and with ``nodetool enableoldprotocolversions`` and ``nodetool disableoldprotocolversions`` during upgrades.
+- To identify clients sending too many requests.
+- To find if SSL is enabled during the migration to and from ssl.
+
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the current configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the virtual table presently.
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_enabled | false
+ auto_snapshot | true
+ automatic_sstable_upgrade | false
+ cluster_name | Test Cluster
+ enable_transient_replication | false
+ hinted_handoff_enabled | true
+ hints_directory | /home/ec2-user/cassandra/data/hints
+ incremental_backups | false
+ initial_token | null
+ ...
+ ...
+ ...
+ rpc_address | localhost
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ (224 rows)
+
+
+The ``settings`` table can be really useful if yaml file has been changed since startup and dont know running configuration, or to find if they have been modified via jmx/nodetool or virtual tables.
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+The ``sstable_tasks`` could be used to get information about running tasks. It lists following columns.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+---------+------+----------+-------+------
+
+
+As an example a query of ``total-progress`` gives the remaining time for a task.
+
+Other Virtual Tables
+********************
+
+Some examples of using other virtual tables are as follows.
+
+::
Review comment:
while it can be used to measure read performance, its value is more in identifying which table it is thats being queried or tables with the worst latency. When we look at graphs we often see spikes in latency or count but it can be difficult to narrow down which queries it is thats causing those spikes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379497736
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings table`` is rather useful and lists all the configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the
+virtual table presently. A total of 224 settings get listed presently.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -----------------------------------------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_audit_logs_dir |
+ /home/ec2-user/cassandra/logs/audit/
+ audit_logging_options_enabled |
+ false
+ audit_logging_options_excluded_categories |
+ audit_logging_options_excluded_keyspaces |
+ system,system_schema,system_virtual_schema
+ audit_logging_options_excluded_users |
+ audit_logging_options_included_categories |
+ audit_logging_options_included_keyspaces |
+ audit_logging_options_included_users |
+ audit_logging_options_logger |
+ BinAuditLogger
+ authenticator |
+ AllowAllAuthenticator
+ authorizer |
+ AllowAllAuthorizer
+ auto_bootstrap |
+ true
+ auto_snapshot |
+ true
+ automatic_sstable_upgrade |
+ false
+ back_pressure_enabled |
+ false
+ back_pressure_strategy |
+ org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}
+ broadcast_address |
+ null
+ broadcast_rpc_address |
+ null
+ cluster_name |
+ Test Cluster
+ column_index_cache_size_in_kb |
+ 2
+ column_index_size_in_kb |
+ 64
+ commit_failure_policy |
+ stop
+ commitlog_compression |
+ null
+ commitlog_directory |
+ /home/ec2-user/cassandra/data/commitlog
+ commitlog_max_compression_buffers_in_pool |
+ 3
+ commitlog_periodic_queue_size |
+ -1
+ commitlog_segment_size_in_mb |
+ 32
+ commitlog_sync |
+ periodic
+ commitlog_sync_batch_window_in_ms |
+ NaN
+ commitlog_sync_group_window_in_ms |
+ NaN
+ commitlog_sync_period_in_ms |
+ 10000
+ commitlog_total_space_in_mb |
+ 2556
+ compaction_large_partition_warning_threshold_mb |
+ 100
+ compaction_throughput_mb_per_sec |
+ 16
+ concurrent_compactors |
+ 2
+ concurrent_counter_writes |
+ 32
+ concurrent_materialized_view_builders |
+ 1
+ concurrent_materialized_view_writes |
+ 32
+ concurrent_reads |
+ 32
+ concurrent_replicates |
+ null
+ concurrent_validations |
+ 2147483647
+ concurrent_writes |
+ 32
+ credentials_update_interval_in_ms |
+ -1
+ credentials_validity_in_ms |
+ 2000
+ cross_node_timeout |
+ false
+ data_file_directories |
+ [/home/ec2-user/cassandra/data/data]
+ diagnostic_events_enabled |
+ false
+ disk_access_mode |
+ mmap
+ disk_failure_policy |
+ stop
+ enable_materialized_views |
+ false
+ enable_sasi_indexes |
+ false
+ enable_scripted_user_defined_functions |
+ false
+ enable_transient_replication |
+ false
+ enable_user_defined_functions |
+ false
+ enable_user_defined_functions_threads |
+ true
+ endpoint_snitch |
+ SimpleSnitch
+ file_cache_round_up |
+ false
+ file_cache_size_in_mb |
+ 251
+ full_query_logging_options |
+ FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true,
+ max_queue_weight=268435456, max_log_size=17179869184}
+ gc_log_threshold_in_ms |
+ 200
+ gc_warn_threshold_in_ms |
+ 1000
+ hinted_handoff_disabled_datacenters |
+ []
+ hinted_handoff_enabled |
+ true
+ hinted_handoff_throttle_in_kb |
+ 1024
+ hints_compression |
+ null
+ hints_directory |
+ /home/ec2-user/cassandra/data/hints
+ hints_flush_period_in_ms |
+ 10000
+ ideal_consistency_level |
+ null
+ incremental_backups |
+ false
+ initial_token |
+ null
+ inter_dc_stream_throughput_outbound_megabits_per_sec |
+ 200
+ inter_dc_tcp_nodelay |
+ false
+ internode_application_receive_queue_capacity_in_bytes |
+ 4194304
+ internode_application_receive_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+ internode_application_receive_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_application_send_queue_capacity_in_bytes |
+ 4194304
+ internode_application_send_queue_reserve_endpoint_capacity_in_bytes |
+ 134217728
+
+
+ internode_application_send_queue_reserve_global_capacity_in_bytes |
+ 536870912
+ internode_authenticator |
+ null
+ internode_compression |
+ dc
+ listen_address |
+ localhost
+ listen_interface |
+ null
+ listen_on_broadcast_address |
+ false
+ max_concurrent_automatic_sstable_upgrades |
+ 1
+ max_hint_window_in_ms |
+ 10800000
+ memtable_allocation_type |
+ heap_buffers
+ min_free_space_per_drive_in_mb |
+ 50
+ native_transport_allow_older_protocols |
+ true
+ native_transport_flush_in_batches_legacy |
+ false
+ native_transport_frame_block_size_in_kb | 32
+ native_transport_idle_timeout_in_ms | 0
+ native_transport_max_concurrent_connections | -1
+ native_transport_max_concurrent_connections_per_ip | -1
+ native_transport_max_concurrent_requests_in_bytes | 105277030
+ native_transport_max_concurrent_requests_in_bytes_per_ip | 26319257
+ native_transport_max_frame_size_in_mb | 256
+ native_transport_max_threads | 128
+ native_transport_port | 9042
+ native_transport_port_ssl | null
+ network_authorizer | AllowAllNetworkAuthorizer
+ num_tokens | 256
+ partitioner | org.apache.cassandra.dht.Murmur3Partitioner
+ prepared_statements_cache_size_mb | null
+ range_request_timeout_in_ms | 10000
+ read_request_timeout_in_ms | 5000
+ repair_command_pool_full_strategy | queue
+ repair_command_pool_size | 2147483647
+ repair_session_max_tree_depth | 20
+ repair_session_space_in_mb | 62
+ repaired_data_tracking_for_partition_reads_enabled | false
+ repaired_data_tracking_for_range_reads_enabled | false
+ report_unconfirmed_repaired_data_mismatches | false
+ request_timeout_in_ms | 10000
+ role_manager | CassandraRoleManager
+ rpc_address | localhost
+ rpc_interface | null
+ saved_caches_directory | /home/ec2-user/cassandra/data/saved_caches
+ seed_provider | org.apache.cassandra.locator.SimpleSeedProvider{seeds=127.0.0.1:7000}
+ server_encryption_options_algorithm | null
+ server_encryption_options_protocol | TLS
+ slow_query_log_timeout_in_ms | 500
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ transparent_data_encryption_options_chunk_length_kb | 64
+ transparent_data_encryption_options_cipher | AES/CBC/PKCS5Padding
+ transparent_data_encryption_options_enabled | false
+ transparent_data_encryption_options_iv_length |
+ 16
+ (224 rows)
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details.
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+As no sstable tasks are running initially the ``system_views.sstable_tasks`` table lists 0 rows.
Review comment:
this is very sparse, can we list kind of operations that could be listed here? and maybe include an example of compactions running. Easiest way to get that btw is to `nodetool disableautocompaction` and run a stress tool for a bit, calling `nodetool flush` frequently to generate lots of sstables. Then `nodetool enableautocompaction` and see them listed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] Deepak-Vohra commented on issue #402: New/virtualtables
Posted by GitBox <gi...@apache.org>.
Deepak-Vohra commented on issue #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#issuecomment-590551266
Thanks @clohfink. @rustyrazorblade Virtual tables is ready to be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on issue #402: New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on issue #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#issuecomment-586409104
Made almost all changes suggested by @clohfink. Or indicated why a change is not made.
Please review and suggest if more changes are needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379513902
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
Review comment:
Not even sure we should say thats different then regular tables since many system tables use LocalPartitioner and are non-replicated. But we should definitely highlight that they these tables are local only and non distributed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379487054
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
Review comment:
like above mbeans/jmx things are not exposed. The theory is eventually this can replace JMX but it currently does not act as a proxy for JMX.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r382795469
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,337 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose metrics through CQL
+- Expose YAML configuration information
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only, but it is likely to change
+- Virtual tables are not replicated
+- Virtual tables are local only and non distributed
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use ``LocalPartitioner``. Since a virtual table is not replicated the partitioner sorts in order of partition keys instead of by their hash.
+- Making advanced queries with ``ALLOW FILTERING`` and aggregation functions may be used with virtual tables even though in normal tables we dont recommend it
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces:
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables.
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations initially though some of these could change such as:
+
+- Cannot alter or drop virtual keyspaces or tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support ``DELETE`` statements
+- Cannot ``CREATE TRIGGER`` against a virtual table
+- Conditional ``BATCH`` statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional ``BATCH`` statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces; but may run aggregate functions on select
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ coordinator_scans clients tombstones_scanned internode_inbound
+ disk_usage sstable_tasks live_scanned caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in ``system_views`` virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name:
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+Some examples of how ``clients`` can be used are:
+
+- To find applications using old incompatible versions of drivers before upgrading and with ``nodetool enableoldprotocolversions`` and ``nodetool disableoldprotocolversions`` during upgrades.
+- To identify clients sending too many requests.
+- To find if SSL is enabled during the migration to and from ssl.
+
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table:
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
+
+Caches Virtual Table
+********************
+The ``caches`` virtual table lists information about the caches. The four caches presently created are chunks, counters, keys and rows. A query on the ``caches`` virtual table returns the following details:
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.caches;
+ name | capacity_bytes | entry_count | hit_count | hit_ratio | recent_hit_rate_per_second | recent_request_rate_per_second | request_count | size_bytes
+ ---------+----------------+-------------+-----------+-----------+----------------------------+--------------------------------+---------------+------------
+ chunks | 229638144 | 29 | 166 | 0.83 | 5 | 6 | 200 | 475136
+ counters | 26214400 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+ keys | 52428800 | 14 | 124 | 0.873239 | 4 | 4 | 142 | 1248
+ rows | 0 | 0 | 0 | NaN | 0 | 0 | 0 | 0
+
+ (4 rows)
+
+Settings Virtual Table
+**********************
+The ``settings`` table is rather useful and lists all the current configuration settings from the ``cassandra.yaml``. The encryption options are overridden to hide the sensitive truststore information or passwords. The configuration settings however cannot be set using DML on the virtual table presently:
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.settings;
+
+ name | value
+ -------------------------------------+--------------------
+ allocate_tokens_for_keyspace | null
+ audit_logging_options_enabled | false
+ auto_snapshot | true
+ automatic_sstable_upgrade | false
+ cluster_name | Test Cluster
+ enable_transient_replication | false
+ hinted_handoff_enabled | true
+ hints_directory | /home/ec2-user/cassandra/data/hints
+ incremental_backups | false
+ initial_token | null
+ ...
+ ...
+ ...
+ rpc_address | localhost
+ ssl_storage_port | 7001
+ start_native_transport | true
+ storage_port | 7000
+ stream_entire_sstables | true
+ (224 rows)
+
+
+The ``settings`` table can be really useful if yaml file has been changed since startup and dont know running configuration, or to find if they have been modified via jmx/nodetool or virtual tables.
+
+
+Thread Pools Virtual Table
+**************************
+
+The ``thread_pools`` table lists information about all thread pools. Thread pool information includes active tasks, active tasks limit, blocked tasks, blocked tasks all time, completed tasks, and pending tasks. A query on the ``thread_pools`` returns following details:
+
+::
+
+ cqlsh:system_views> select * from system_views.thread_pools;
+
+ name | active_tasks | active_tasks_limit | blocked_tasks | blocked_tasks_all_time | completed_tasks | pending_tasks
+ ------------------------------+--------------+--------------------+---------------+------------------------+-----------------+---------------
+ AntiEntropyStage | 0 | 1 | 0 | 0 | 0 | 0
+ CacheCleanupExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ CompactionExecutor | 0 | 2 | 0 | 0 | 881 | 0
+ CounterMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ GossipStage | 0 | 1 | 0 | 0 | 0 | 0
+ HintsDispatcher | 0 | 2 | 0 | 0 | 0 | 0
+ InternalResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ MemtableFlushWriter | 0 | 2 | 0 | 0 | 1 | 0
+ MemtablePostFlush | 0 | 1 | 0 | 0 | 2 | 0
+ MemtableReclaimMemory | 0 | 1 | 0 | 0 | 1 | 0
+ MigrationStage | 0 | 1 | 0 | 0 | 0 | 0
+ MiscStage | 0 | 1 | 0 | 0 | 0 | 0
+ MutationStage | 0 | 32 | 0 | 0 | 0 | 0
+ Native-Transport-Requests | 1 | 128 | 0 | 0 | 130 | 0
+ PendingRangeCalculator | 0 | 1 | 0 | 0 | 1 | 0
+ PerDiskMemtableFlushWriter_0 | 0 | 2 | 0 | 0 | 1 | 0
+ ReadStage | 0 | 32 | 0 | 0 | 13 | 0
+ Repair-Task | 0 | 2147483647 | 0 | 0 | 0 | 0
+ RequestResponseStage | 0 | 2 | 0 | 0 | 0 | 0
+ Sampler | 0 | 1 | 0 | 0 | 0 | 0
+ SecondaryIndexManagement | 0 | 1 | 0 | 0 | 0 | 0
+ ValidationExecutor | 0 | 2147483647 | 0 | 0 | 0 | 0
+ ViewBuildExecutor | 0 | 1 | 0 | 0 | 0 | 0
+ ViewMutationStage | 0 | 32 | 0 | 0 | 0 | 0
+
+(24 rows)
+
+Internode Inbound Messaging Virtual Table
+*****************************************
+
+The ``internode_inbound`` virtual table is for the internode inbound messaging. Initially no internode inbound messaging may get listed. In addition to the address, port, datacenter and rack information includes corrupt frames recovered, corrupt frames unrecovered, error bytes, error count, expired bytes, expired count, processed bytes, processed count, received bytes, received count, scheduled bytes, scheduled count, throttled count, throttled nanos, using bytes, using reserve bytes. A query on the ``internode_inbound`` returns following details:
+
+::
+
+ cqlsh:system_views> SELECT * FROM system_views.internode_inbound;
+ address | port | dc | rack | corrupt_frames_recovered | corrupt_frames_unrecovered |
+ error_bytes | error_count | expired_bytes | expired_count | processed_bytes |
+ processed_count | received_bytes | received_count | scheduled_bytes | scheduled_count | throttled_count | throttled_nanos | using_bytes | using_reserve_bytes
+ ---------+------+----+------+--------------------------+----------------------------+-
+ ----------
+ (0 rows)
+
+SSTables Tasks Virtual Table
+****************************
+
+The ``sstable_tasks`` could be used to get information about running tasks. It lists following columns:
+
+::
+
+ cqlsh:system_views> SELECT * FROM sstable_tasks;
+ keyspace_name | table_name | task_id | kind | progress | total | unit
+ ---------------+------------+--------------------------------------+------------+----------+----------+-------
+ basic | wide2 | c3909740-cdf7-11e9-a8ed-0f03de2d9ae1 | compaction | 60418761 | 70882110 | bytes
+ basic | wide2 | c7556770-cdf7-11e9-a8ed-0f03de2d9ae1 | compaction | 2995623 | 40314679 | bytes
+
+
+As another example, a query ``SELECT (total-progress) FROM sstable_tasks;`` gives the remaining time for a task.
Review comment:
thats not a valid query (there is no total-progress column) and theres no ability to estimate time remaining in the virtual table
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379512776
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
Review comment:
I think we should mention here about making advanced queries with ALLOW FILTERING and aggregation functions are safe to use even though in normal tables we dont recommend it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] dvohra commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
dvohra commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379568407
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
Review comment:
Added
- the partitioner sorts in order of partition keys instead of by their hash
- tables are local only and non distributed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] clohfink commented on a change in pull request #402:
New/virtualtables
Posted by GitBox <gi...@apache.org>.
clohfink commented on a change in pull request #402: New/virtualtables
URL: https://github.com/apache/cassandra/pull/402#discussion_r379494446
##########
File path: doc/source/new/virtualtables.rst
##########
@@ -0,0 +1,586 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+Virtual Tables
+--------------
+
+Apache Cassandra 4.0 implements virtual tables (`CASSANDRA-7622
+<https://issues.apache.org/jira/browse/CASSANDRA-7622>`_).
+
+Definition
+^^^^^^^^^^
+
+A virtual table is a table that is backed by an API instead of data explicitly managed and stored as SSTables. Apache Cassandra 4.0 implements a virtual keyspace interface for virtual tables. Virtual tables are specific to each node.
+
+Objective
+^^^^^^^^^
+
+A virtual table could have several uses including:
+
+- Expose JMX data/metrics through CQL
+- Expose YAML configuration information
+- Expose other MBean data
+
+How are Virtual Tables different from regular tables?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces are quite different from regular tables and keyspaces respectively such as:
+
+- Virtual tables are read-only
+- Virtual tables are not replicated
+- Virtual tables have no associated SSTables
+- Consistency level of the queries sent virtual tables are ignored
+- Virtual tables are managed by Cassandra and a user cannot run DDL to create new virtual tables or DML to modify existing virtual tables
+- Virtual tables are created in special keyspaces and not just any keyspace
+- All existing virtual tables use LocalPartitioner
+
+Virtual Keyspaces
+^^^^^^^^^^^^^^^^^
+
+Apache Cassandra 4.0 has added two new keyspaces for virtual tables: ``system_virtual_schema`` and ``system_views``. Run the following command to list the keyspaces.
+
+::
+
+ cqlsh> DESC KEYSPACES;
+ system_schema system system_distributed system_virtual_schema
+ system_auth system_traces system_views
+
+The ``system_virtual_schema keyspace`` contains schema information on virtual tables. The ``system_views`` keyspace contains the actual virtual tables. The virtual keyspaces metadata is not exposed through ``DESCRIBE`` statement, which returns an error message:
+
+::
+
+ cqlsh> DESCRIBE KEYSPACE system_views
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh> DESCRIBE KEYSPACE system_virtual_schema;
+ 'NoneType' object has no attribute 'export_for_schema'
+ cqlsh>
+
+Virtual Table Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables and virtual keyspaces have some limitations such as:
+
+- Cannot alter virtual keyspaces
+- Cannot drop virtual keyspaces
+- Cannot alter virtual tables
+- Cannot drop virtual tables
+- Cannot truncate virtual tables
+- Expiring columns are not supported by virtual tables
+- Conditional updates are not supported by virtual tables
+- Cannot create tables in virtual keyspaces
+- Cannot perform any operations against virtual keyspace
+- Secondary indexes are not supported on virtual tables
+- Cannot create functions in virtual keyspaces
+- Cannot create types in virtual keyspaces
+- Materialized views are not supported on virtual tables
+- Virtual tables don't support DELETE statements
+- Cannot CREATE TRIGGER against a virtual table
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot include a virtual table statement in a logged batch
+- Mutations for virtual and regular tables cannot exist in the same batch
+- Conditional BATCH statements cannot include mutations for virtual tables
+- Cannot create aggregates in virtual keyspaces
+
+Listing and Describing Virtual Tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual tables in a virtual keyspace may however be listed with ``DESC TABLES``. The ``system_views`` virtual keyspace tables include the following:
+
+::
+
+ cqlsh> USE system_views;
+ cqlsh:system_views> DESC TABLES;
+ clients internode_inbound
+ disk_usage sstable_tasks caches
+ local_writes max_partition_size local_reads
+ coordinator_writes internode_outbound thread_pools
+ local_scans coordinator_reads settings
+
+Some of the salient virtual tables in system_views virtual keyspace are described in Table 1.
+
+Table 1 : Virtual Tables in system_views
+
++------------------+---------------------------------------------------+
+|Virtual Table | Description |
++------------------+---------------------------------------------------+
+| clients |Lists information about all connected clients. |
++------------------+---------------------------------------------------+
+| disk_usage |Disk usage including disk_space, keyspace_name, |
+| |and table_name by system keyspaces. |
++------------------+---------------------------------------------------+
+| local_writes |A table metric for local writes |
+| |including count, keyspace_name, |
+| |max, median, per_second, and |
+| |table_name. |
++------------------+---------------------------------------------------+
+| caches |Displays the general cache information including |
+| |cache name, capacity_bytes, entry_count, hit_count,|
+| |hit_ratio double, recent_hit_rate_per_second, |
+| |recent_request_rate_per_second, request_count, and |
+| |size_bytes. |
++------------------+---------------------------------------------------+
+| local_reads |A table metric for local reads information. |
++------------------+---------------------------------------------------+
+| sstable_tasks |Lists currently running tasks such as compactions |
+| |and upgrades on SSTables. |
++------------------+---------------------------------------------------+
+|internode_inbound |Lists information about the inbound |
+| |internode messaging. |
++------------------+---------------------------------------------------+
+| thread_pools |Lists metrics for each thread pool. |
++------------------+---------------------------------------------------+
+| settings |Displays configuration settings in cassandra.yaml. |
++------------------+---------------------------------------------------+
+|max_partition_size|A table metric for maximum partition size. |
++------------------+---------------------------------------------------+
+|internode_outbound|Information about the outbound internode messaging.|
+| | |
++------------------+---------------------------------------------------+
+
+We shall discuss some of the virtual tables in more detail next.
+
+Clients Virtual Table
+*********************
+
+The ``clients`` virtual table lists all active connections (connected clients) including their ip address, port, connection stage, driver name, driver version, hostname, protocol version, request count, ssl enabled, ssl protocol and user name. A query on the ``clients`` table returns the following details.
+
+::
+
+ cqlsh:system_views> select * from system_views.clients;
+ address | port | connection_stage | driver_name | driver_version | hostname | protocol_version | request_count | ssl_cipher_suite | ssl_enabled | ssl_protocol | username
+ -----------+-------+------------------+-------------+----------------+-----------+------------------+---------------+------------------+-------------+--------------+-----------
+ 127.0.0.1 | 50628 | ready | null | null | localhost | 4 | 55 | null | False | null | anonymous
+ 127.0.0.1 | 50630 | ready | null | null | localhost | 4 | 70 | null | False | null | anonymous
+
+ (2 rows)
+
+The virtual tables may be described with ``DESCRIBE`` statement. The DDL listed however cannot be run to create a virtual table. As an example describe the ``system_views.clients`` virtual table.
+
+::
+
+ cqlsh:system_views> DESC TABLE system_views.clients;
+ CREATE TABLE system_views.clients (
+ address inet,
+ connection_stage text,
+ driver_name text,
+ driver_version text,
+ hostname text,
+ port int,
+ protocol_version int,
+ request_count bigint,
+ ssl_cipher_suite text,
+ ssl_enabled boolean,
+ ssl_protocol text,
+ username text,
+ PRIMARY KEY (address, port)) WITH CLUSTERING ORDER BY (port ASC)
+ AND compaction = {'class': 'None'}
+ AND compression = {};
Review comment:
We can leave these in here for now but with the driver fixes this changes quite a bit to be a commented out block with less of the other table options. If that gets in 4.0 GA we should circle around and replace this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org