You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ignite.apache.org by ab...@apache.org on 2020/08/04 14:38:22 UTC
[ignite] branch IGNITE-7595 updated: add more pages
This is an automated email from the ASF dual-hosted git repository.
abudnikov pushed a commit to branch IGNITE-7595
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/IGNITE-7595 by this push:
new 3abd92a add more pages
3abd92a is described below
commit 3abd92a9acbe8356f5fe557ff4d4bdd5a53a056d
Author: abudnikov <ab...@gridgain.com>
AuthorDate: Tue Aug 4 17:37:07 2020 +0300
add more pages
---
docs/README.adoc | 114 ++++-
docs/_data/toc.yaml | 6 +
docs/_docs/SQL/JDBC/jdbc-client-driver.adoc | 4 +-
docs/_docs/SQL/indexes.adoc | 2 +-
docs/_docs/SQL/sql-tuning.adoc | 457 +++++++++++++++++++++
.../code-snippets/dotnet/PersistenceTuning.cs | 79 ++++
docs/_docs/code-snippets/java/pom.xml | 6 +
.../apache/ignite/snippets/DiskCompression.java | 41 ++
.../apache/ignite/snippets/IgnitePersistence.java | 50 ++-
.../apache/ignite/snippets/PersistenceTuning.java | 93 +++++
.../main/java/org/apache/ignite/snippets/WAL.java | 30 ++
docs/_docs/code-snippets/xml/disk-compression.xml | 43 ++
.../_docs/code-snippets/xml/persistence-tuning.xml | 65 +++
.../configuring-caches/configuration-overview.adoc | 2 +-
docs/_docs/events/events.adoc | 2 +
docs/_docs/images/set-streaming.png | Bin 0 -> 56005 bytes
.../kubernetes/generic-configuration.adoc | 2 +-
docs/_docs/monitoring-metrics/intro.adoc | 2 +-
docs/_docs/monitoring-metrics/metrics.adoc | 2 +-
docs/_docs/monitoring-metrics/system-views.adoc | 13 +-
docs/_docs/persistence/disk-compression.adoc | 48 +++
docs/_docs/persistence/native-persistence.adoc | 32 +-
docs/_docs/persistence/persistence-tuning.adoc | 244 +++++++++++
docs/_docs/sql-reference/ddl.adoc | 2 +-
docs/_docs/sql-reference/operational-commands.adoc | 4 +-
docs/_docs/sql-reference/sql-conformance.adoc | 4 +-
docs/_docs/sqlline.adoc | 211 ++++++++++
docs/_docs/thin-clients/java-thin-client.adoc | 2 -
docs/_docs/thin-clients/nodejs-thin-client.adoc | 2 +-
docs/_docs/thin-clients/php-thin-client.adoc | 2 +-
docs/_docs/thin-clients/python-thin-client.adoc | 4 +-
31 files changed, 1518 insertions(+), 50 deletions(-)
diff --git a/docs/README.adoc b/docs/README.adoc
index e2cb63e..f8c477c 100644
--- a/docs/README.adoc
+++ b/docs/README.adoc
@@ -54,7 +54,117 @@ $ docker run -v "$PWD:/srv/jekyll" -p 4000:4000 jekyll/jekyll:latest jekyll s
Open `http://localhost:4000/docs[window=_blank]` in your browser.
-
== How to Contribute
-TBD
+If you want to contribute to the documentation, add or modify the relevant page in the `docs/_docs` directory.
+This directory contains all .adoc files, which are rendered into HTML.
+
+
+Because we use asciidoc for documentation, get familiar with the following resources:
+
+* Get familiar with the asciidoc format: https://asciidoctor.org/docs/user-manual/. You don’t have to read the entire manual. Search through it when you want to learn how to create a numbered list, or insert an image, or use italics.
+* Please read the link:https://asciidoctor.org/docs/asciidoc-recommended-practices:[AsciiDoc Recommended Practices] and try to adhere to those when editing the source files.
+
+
+
+=== Table of content
+
+The table of content is defined in the `_data/toc.yaml` file.
+If you want to add a new page, make sure to update the TOC.
+
+
+=== Links to other sections in the docs
+All .adoc files are located in the "docs/_docs" directory.
+Any link to the files within the directory must be relative to that directory.
+Remove the file extension (.adoc).
+
+For example:
+[source, adoc]
+----
+link:persistence/native-persistence[Native Persistence]
+----
+
+This is a link to the Native Persistence page.
+
+=== Links to external resources
+
+When referencing an external resource, make the link to open in a new window by adding the `window=_blank` attribute:
+
+[source, adoc]
+----
+link:https://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#SunJSSE_Protocols[Supported protocols,window=_blank]
+----
+
+
+
+=== Tabs
+
+We use custom syntax to insert tabs. Tabs are used to provide code samples for different programming languages.
+
+Tabs are defined by the `tabs` block:
+```
+[tabs]
+--
+individual tabs are defined here
+--
+```
+
+Each tab is defined by the 'tab' directive:
+
+```
+tab:tab_name[]
+```
+
+where `tab_name` is the title of the tab.
+
+The content of the tab is everything that is given between the tab title, and the next tab or the end of the block.
+
+```asciidoc
+[tabs]
+--
+tab:XML[]
+
+The content of the XML tab goes here
+
+tab:Java[]
+
+The content of the Java tab is here
+
+tab:C#/.NET[]
+
+tab:C++[unsupported]
+
+--
+```
+
+=== Code Snippets
+
+Compilable source code files are located in the `docs/_docs/code-snippets/{language}` folders.
+
+
+To add a code snippet to a page, follow these steps:
+
+* Create a file in the code snippets directory, e.g. _docs/code-snippets/java/org/apache/ignite/snippets/JavaThinClient.java
+
+* Enclose the piece of code you want to include within named tags (see https://asciidoctor.org/docs/user-manual/#by-tagged-regions). Give the tag a self-evident name.
+For example:
++
+```
+[source, java]
+----
+// tag::clientConnection[]
+ClientConfiguration cfg = new ClientConfiguration().setAddresses("127.0.0.1:10800");
+try (IgniteClient client = Ignition.startClient(cfg)) {
+ ClientCache<Integer, String> cache = client.cache("myCache");
+ // get data from the cache
+}
+// end::clientConnection[]
+----
+```
+
+* Include the tag in the adoc file:
++
+[source, adoc,subs="macros"]
+----
+\include::{javaCodeDir}/JavaThinClient.java[tag=clientConnection,indent=0]
+----
diff --git a/docs/_data/toc.yaml b/docs/_data/toc.yaml
index 2665b18..d3542e4 100644
--- a/docs/_data/toc.yaml
+++ b/docs/_data/toc.yaml
@@ -79,6 +79,10 @@
url: /persistence/swap
- title: Implementing Custom Cache Store
url: /persistence/custom-cache-store
+ - title: Disk Compression
+ url: /persistence/disk-compression
+ - title: Tuning Persistence
+ url: /persistence/persistence-tuning
- title: Baseline Topology
url: /baseline-topology
- title: Data Rebalancing
@@ -205,6 +209,8 @@
url: /restapi
- title: Control Script
url: /control-script
+- title: SQLLine
+ url: /sqlline
# - title: Capacity Planning
# url: /capacity-planning
diff --git a/docs/_docs/SQL/JDBC/jdbc-client-driver.adoc b/docs/_docs/SQL/JDBC/jdbc-client-driver.adoc
index 5dab3c4..34834c7 100644
--- a/docs/_docs/SQL/JDBC/jdbc-client-driver.adoc
+++ b/docs/_docs/SQL/JDBC/jdbc-client-driver.adoc
@@ -1,11 +1,11 @@
= JDBC Client Driver
:javaFile: {javaCodeDir}/JDBCClientDriver.java
-JDBC Client Driver interacts with the cluster by means of a link:getting-started/concepts#clustering-servers-and-clients[client node].
+JDBC Client Driver interacts with the cluster by means of a client node.
== JDBC Client Driver
-The JDBC Client Driver connects to the cluster by using a link:getting-started/concepts#clustering-servers-and-clients[client node] connection. You must provide a complete Spring XML configuration as part of the JDBC connection string, and copy all the JAR files mentioned below to the classpath of your application or SQL tool:
+The JDBC Client Driver connects to the cluster by using a lclient node connection. You must provide a complete Spring XML configuration as part of the JDBC connection string, and copy all the JAR files mentioned below to the classpath of your application or SQL tool:
- All the JARs under `{IGNITE_HOME}\libs` directory.
- All the JARs under `{IGNITE_HOME}\ignite-indexing` and `{IGNITE_HOME}\ignite-spring` directories.
diff --git a/docs/_docs/SQL/indexes.adoc b/docs/_docs/SQL/indexes.adoc
index f837435..b764ff5 100644
--- a/docs/_docs/SQL/indexes.adoc
+++ b/docs/_docs/SQL/indexes.adoc
@@ -226,7 +226,7 @@ Refer to the link:sql-reference/ddl#create-index[CREATE INDEX] section.
Proper index inline size can help speed up queries on indexed fields.
//For primitive types and BinaryObjects, Ignite uses a predefined inline index size
-Refer to the dedicated section in the link:perf-troubleshooting-guide/sql-tuning#increasing-index-inline-size[SQL Tuning guide] for the information on how to choose a proper inline size.
+Refer to the dedicated section in the link:SQL/sql-tuning#increasing-index-inline-size[SQL Tuning guide] for the information on how to choose a proper inline size.
In most cases, you will only need to set the inline size for indexes on variable-length fields, such as strings or arrays.
The default value is 10.
diff --git a/docs/_docs/SQL/sql-tuning.adoc b/docs/_docs/SQL/sql-tuning.adoc
new file mode 100644
index 0000000..72f22d4
--- /dev/null
+++ b/docs/_docs/SQL/sql-tuning.adoc
@@ -0,0 +1,457 @@
+= SQL Performance Tuning
+
+This article outlines basic and advanced optimization techniques for Ignite SQL queries. Some of the sections are also useful for debugging and troubleshooting.
+
+
+== Using the EXPLAIN Statement
+
+Ignite supports the `EXPLAIN` statement, which could be used to read the execution plan of a query.
+Use this command to analyse your queries for possible optimization.
+Note that the plan will contain multiple rows: the last one will contain a query for the reducing side (usually your application), others are for map nodes (usually server nodes).
+Read the link:SQL/sql-introduction#distributed-queries[Distributed Queries] section to learn how queries are executed in Ignite.
+
+[source,sql]
+----
+EXPLAIN SELECT name FROM Person WHERE age = 26;
+----
+
+The execution plan is generated by H2 as described link:http://www.h2database.com/html/performance.html#explain_plan[here, window=_blank].
+
+== OR Operator and Selectivity
+
+//*TODO*: is this still valid?
+
+If a query contains an `OR` operator, then indexes may not be used as expected depending on the complexity of the query.
+For example, for the query `select name from Person where gender='M' and (age = 20 or age = 30)`, an index on the `gender` field will be used instead of an index on the `age` field, although the latter is a more selective index.
+As a workaround for this issue, you can rewrite the query with `UNION ALL` (notice that `UNION` without `ALL` will return `DISTINCT` rows, which will change the query semantics and will further penalize your query performance):
+
+[source,sql]
+----
+SELECT name FROM Person WHERE gender='M' and age = 20
+UNION ALL
+SELECT name FROM Person WHERE gender='M' and age = 30
+----
+
+== Avoid Having Too Many Columns
+
+Avoid having too many columns in the result set of a `SELECT` query. Due to limitations of the H2 query parser, queries with 100+ columns may perform worse than expected.
+
+== Lazy Loading
+
+By default, Ignite attempts to load the whole result set to memory and send it back to the query initiator (which is usually your application).
+This approach provides optimal performance for queries of small or medium result sets.
+However, if the result set is too big to fit in the available memory, it can lead to prolonged GC pauses and even `OutOfMemoryError` exceptions.
+
+To minimize memory consumption, at the cost of a moderate performance hit, you can load and process the result sets lazily by passing the `lazy` parameter to the JDBC and ODBC connection strings or use a similar method available for Java, .NET, and C++ APIs:
+
+[tabs]
+--
+
+tab:Java[]
+[source,java]
+----
+SqlFieldsQuery query = new SqlFieldsQuery("SELECT * FROM Person WHERE id > 10");
+
+// Result set will be loaded lazily.
+query.setLazy(true);
+----
+tab:JDBC[]
+[source,sql]
+----
+jdbc:ignite:thin://192.168.0.15?lazy=true
+----
+tab:C#/.NET[]
+[source,csharp]
+----
+var query = new SqlFieldsQuery("SELECT * FROM Person WHERE id > 10")
+{
+ // Result set will be loaded lazily.
+ Lazy = true
+};
+----
+tab:C++[]
+--
+
+////
+*TODO* Add tabs for ODBC and other programming languages - C# and C++
+////
+
+== Querying Colocated Data
+
+When Ignite executes a distributed query, it sends sub-queries to individual cluster nodes to fetch the data and groups the results on the reducer node (usually your application).
+If you know in advance that the data you are querying is link:data-modeling/affinity-collocation[colocated] by the `GROUP BY` condition, you can use `SqlFieldsQuery.collocated = true` to tell the SQL engine to do the grouping on the remote nodes.
+This will reduce network traffic between the nodes and query execution time.
+When this flag is set to `true`, the query is executed on individual nodes first and the results are sent to the reducer node for final calculation.
+
+Consider the following example, in which we assume that the data is colocated by `department_id` (in other words, the `department_id` field is configured as the affinity key).
+
+[source,sql]
+----
+SELECT SUM(salary) FROM Employee GROUP BY department_id
+----
+
+Because of the nature of the SUM operation, Ignite sums up the salaries across the elements stored on individual nodes, and then sends these sums to the reducer node where the final result are calculated.
+This operation is already distributed, and enabling the `collocated` flag only slightly improves performance.
+
+Let's take a slightly different example:
+
+[source,sql]
+----
+SELECT AVG(salary) FROM Employee GROUP BY department_id
+----
+
+In this example, Ignite has to fetch all (`salary`, `department_id`) pairs to the reducer node and calculate the results there.
+However, if employees are colocated by the `department_id` field, i.e. employee data for the same department is stored on the same node, setting `SqlFieldsQuery.collocated = true` reduces query execution time because Ignite calculates the averages for each department on the individual nodes and sends the results to the reducer node for final calculation.
+
+
+== Enforcing Join Order
+
+When this flag is set, the query optimizer will not reorder tables in joins.
+In other words, the order in which joins are applied during query execution will be the same as specified in the query.
+Without this flag, the query optimizer can reorder joins to improve performance.
+However, sometimes it might make an incorrect decision.
+This flag helps to control and explicitly specify the order of joins instead of relying on the optimizer.
+
+Consider the following example:
+
+[source, sql]
+----
+SELECT * FROM Person p
+JOIN Company c ON p.company = c.name where p.name = 'John Doe'
+AND p.age > 20
+AND p.id > 5000
+AND p.id < 100000
+AND c.name NOT LIKE 'O%';
+----
+
+This query contains a join between two tables: `Person` and `Company`.
+To get the best performance, we should understand which join will return the smallest result set.
+The table with the smaller result set size should be given first in the join pair.
+To get the size of each result set, let's test each part.
+
+.Q1:
+[source, sql]
+----
+SELECT count(*)
+FROM Person p
+where
+p.name = 'John Doe'
+AND p.age > 20
+AND p.id > 5000
+AND p.id < 100000;
+----
+
+.Q2:
+[source, sql]
+----
+SELECT count(*)
+FROM Company c
+where
+c.name NOT LIKE 'O%';
+----
+
+After running Q1 and Q2, we can get two different outcomes:
+
+Case 1:
+[cols="1,1",opts="stretch,autowidth",stripes=none]
+|===
+|Q1 | 30000
+|Q2 |100000
+|===
+
+Q2 returns more entries than Q1.
+In this case, we don't need to modify the original query, because smaller subset has already been located on the left side of the join.
+
+Case 2:
+[cols="1,1",opts="stretch,autowidth",stripes=none]
+|===
+|Q1 | 50000
+|Q2 |10000
+|===
+
+Q1 returns more entries than Q2. So we need to change the initial query as follows:
+
+[source, sql]
+----
+SELECT *
+FROM Company c
+JOIN Person p
+ON p.company = c.name
+where
+p.name = 'John Doe'
+AND p.age > 20
+AND p.id > 5000
+AND p.id < 100000
+AND c.name NOT LIKE 'O%';
+----
+
+The force join order hint can be specified as follows:
+
+* link:SQL/JDBC/jdbc-driver#parameters[JDBC driver connection parameter]
+* link:SQL/ODBC/connection-string-dsn#supported-arguments[ODBC driver connection attribute]
+* If you use link:SQL/sql-api[SqlFieldsQuery] to execute SQL queries, you can set the enforce join order hint by calling the `SqlFieldsQuery.setEnforceJoinOrder(true)` method.
+
+
+== Increasing Index Inline Size
+
+Every entry in the index has a constant size which is calculated during index creation. This size is called _index inline size_.
+Ideally this size should be enough to store full indexed entry in serialized form.
+When values are not fully included in the index, Ignite may need to perform additional data page reads during index lookup, which can impair performance if persistence is enabled.
+
+
+Here is how values are stored in the index:
+
+// the source code block below uses css-styles from the pygments library. If you change the highlighting library, you should change the syles as well.
+[source,java,subs="quotes"]
+----
+[tok-kt]#int#
+0 1 5
+| tag | value |
+[tok-k]#Total: 5 bytes#
+
+[tok-kt]#long#
+0 1 9
+| tag | value |
+[tok-k]#Total: 9 bytes#
+
+[tok-kt]#String#
+0 1 3 N
+| tag | size | UTF-8 value |
+[tok-k]#Total: 3 + string length#
+
+[tok-kt]#POJO (BinaryObject)#
+0 1 5
+| tag | BO hash |
+[tok-k]#Total: 5#
+----
+
+For primitive data types (bool, byte, short, int, etc.), Ignite automatically calculates the index inline size so that the values are included in full.
+For example, for `int` fields, the inline size is 5 (1 byte for the tag and 4 bytes for the value itself). For `long` fields, the inline size is 9 (1 byte for the tag + 8 bytes for the value).
+
+For binary objects, the index includes the hash of each object, which is enough to avoid collisions. The inline size is 5.
+
+For variable length data, indexes include only first several bytes of the value.
+Therefore, when indexing fields with variable-length data, we recommend that you estimate the length of your field values and set the inline size to a value that includes most (about 95%) or all values.
+For example, if you have a `String` field with 95% of the values containing 10 characters or fewer, you can set the inline size for the index on that field to 13.
+
+
+The inline sizes explained above apply to single field indexes.
+However, when you define an index on a field in the value object or on a non-primary key column, Ignite creates a _composite index_ by appending the primary key to the indexed value.
+Therefore, when calculating the inline size for composite indexes, add up the inline size of the primary key.
+
+
+Below is an example of index inline size calculation for a cache where both key and value are complex objects.
+
+[source, java]
+----
+public class Key {
+ @QuerySqlField
+ private long id;
+
+ @QuerySqlField
+ @AffinityKeyMapped
+ private long affinityKey;
+}
+
+public class Value {
+ @QuerySqlField(index = true)
+ private long longField;
+
+ @QuerySqlField(index = true)
+ private int intField;
+
+ @QuerySqlField(index = true)
+ private String stringField; // we suppose that 95% of the values are 10 symbols
+}
+----
+
+The following table summarizes the inline index sizes for the indexes defined in the example above.
+
+[cols="1,1,1,2",opts="stretch,header"]
+|===
+|Index | Kind | Recommended Inline Size | Comment
+
+| (_key)
+|Primary key index
+| 5
+|Inlined hash of a binary object (5)
+
+|(affinityKey, _key)
+|Affinity key index
+|14
+|Inlined long (9) + binary object's hash (5)
+
+|(longField, _key)
+|Secondary index
+|14
+|Inlined long (9) + binary object's hash (5)
+
+|(intField, _key)
+|Secondary index
+|10
+|Inlined int (5) + binary object up to hash (5)
+
+|(stringField, _key)
+|Secondary index
+|18
+|Inlined string (13) + binary object's hash (5) (assuming that the string is {tilde}10 symbols)
+
+|===
+//_
+
+//The inline size for the first two indexes is set via `CacheConfiguration.sqlIndexMaxInlineSize = 29` (because a single property is responsible for two indexes, we set it to the largest value).
+//The inline size for the rest of the indexes is set when you define a corresponding index.
+Note that you will only have to set the inline size for the index on `stringField`. For other indexes, Ignite calculates the inline size automatically.
+
+Refer to the link:SQL/indexes#configuring-index-inline-size[Configuring Index Inline Size] section for the information on how to change the inline size.
+
+You can check the inline size of an existing index in the link:monitoring-metrics/system-views#indexes-view[INDEXES] system view.
+
+[WARNING]
+====
+Note that since Ignite encodes strings to `UTF-8`, some characters use more than 1 byte.
+====
+
+== Query Parallelism
+
+By default, a SQL query is executed in a single thread on each participating node. This approach is optimal for queries returning small result sets involving index search. For example:
+
+[source,sql]
+----
+SELECT * FROM Person WHERE p.id = ?;
+----
+
+Certain queries might benefit from being executed in multiple threads.
+This relates to queries with table scans and aggregations, which is often the case for HTAP and OLAP workloads.
+For example:
+
+[source,sql]
+----
+SELECT SUM(salary) FROM Person;
+----
+
+The number of threads created on a single node for query execution is configured per cache and by default equals 1.
+You can change the value by setting the `CacheConfiguration.queryParallelism` parameter.
+If you create SQL tables using the CREATE TABLE command, you can use a link:configuring-caches/configuration-overview#cache-templates[cache template] to set this parameter.
+
+If a query contains `JOINs`, then all the participating caches must have the same degree of parallelism.
+
+== Index Hints
+
+Index hints are useful in scenarios when you know that one index is more suitable for certain queries than another.
+You can use them to instruct the query optimizer to choose a more efficient execution plan.
+To do this, you can use `USE INDEX(indexA,...,indexN)` statement as shown in the following example.
+
+
+[source,sql]
+----
+SELECT * FROM Person USE INDEX(index_age)
+WHERE salary > 150000 AND age < 35;
+----
+
+
+== Partition Pruning
+
+Partition pruning is a technique that optimizes queries that use affinity keys in the `WHERE` condition.
+When executing such a query, Ignite scans only those partitions where the requested data is stored.
+This reduces query time because the query is sent only to the nodes that store the requested partitions.
+
+In the following example, the employee objects are colocated by the `id` field (if an affinity key is not set
+explicitly then the primary key is used as the affinity key):
+
+
+[source,sql]
+----
+CREATE TABLE employee (id BIGINT PRIMARY KEY, department_id INT, name VARCHAR)
+
+/* This query is sent to the node where the requested key is stored */
+SELECT * FROM employee WHERE id=10;
+
+/* This query is sent to all nodes */
+SELECT * FROM employee WHERE department_id=10;
+----
+
+In the next example, the affinity key is set explicitly and, therefore, will be used to colocate data and direct
+queries to the nodes that keep primary copies of the data:
+
+
+[source,sql]
+----
+CREATE TABLE employee (id BIGINT PRIMARY KEY, department_id INT, name VARCHAR) WITH "AFFINITY_KEY=department_id"
+
+/* This query is sent to all nodes */
+SELECT * FROM employee WHERE id=10;
+
+/* This query is sent to the node where the requested key is stored */
+SELECT * FROM employee WHERE department_id=10;
+----
+
+
+[NOTE]
+====
+Refer to link:data-modeling/affinity-collocation[affinity colocation] page for more details
+on how data gets colocated and how it helps boost performance in distributed storages like Ignite.
+====
+
+== Skip Reducer on Update
+
+When Ignite executes a DML operation, it first fetches all the affected intermediate rows for analysis to the reducer node (usually your application), and only then prepares batches of updated values that will be sent to remote nodes.
+
+This approach might affect performance and saturate the network if a DML operation has to move many entries.
+
+Use this flag as a hint for the SQL engine to do all intermediate rows analysis and updates “in-place” on the server nodes. The hint is supported for JDBC and ODBC connections.
+
+
+[tabs]
+--
+tab:JDBC Connection String[]
+[source,text]
+----
+//jdbc connection string
+jdbc:ignite:thin://192.168.0.15/skipReducerOnUpdate=true
+----
+--
+
+== SQL On-heap Row Cache
+
+Ignite stores data and indexes in its own memory space outside of Java heap. This means that with every data
+access, a part of the data will be copied from the off-heap space to Java heap, potentially deserialized, and kept in
+the heap as long as your application or server node references it.
+
+The SQL on-heap row cache is intended to store hot rows (key-value objects) in Java heap, minimizing resources
+spent for data copying and deserialization. Each cached row refers to an entry in the off-heap region and can be
+invalidated when one of the following happens:
+
+* The master entry stored in the off-heap region is updated or removed.
+* The data page that stores the master entry is evicted from RAM.
+
+The on-heap row cache can be enabled for a specific cache/table (if you use `CREATE TABLE` to create SQL tables and caches, then the parameter can be passed via a link:configuring-caches/configuration-overview#cache-templates[cache template]):
+
+
+[source,xml]
+----
+include::code-snippets/xml/sql-on-heap-cache.xml[tags=ignite-config;!discovery,indent=0]
+----
+
+////
+*TODO* Add tabs for ODBC/JDBC and other programming languages - Java C# and C++
+////
+
+If the row cache is enabled, you might be able to trade RAM for performance. You might get up to a 2x performance increase for some SQL queries and use cases by allocating more RAM for rows caching purposes.
+
+[WARNING]
+====
+[discrete]
+=== SQL On-Heap Row Cache Size
+
+Presently, the cache is unlimited and can occupy as much RAM as allocated to your memory data regions. Make sure to:
+
+* Set the JVM max heap size equal to the total size of all the data regions that store caches for which this on-heap row cache is enabled.
+
+* link:perf-troubleshooting-guide/memory-tuning#java-heap-and-gc-tuning[Tune] JVM garbage collection accordingly.
+====
+
+== Using TIMESTAMP instead of DATE
+
+//TODO: is this still valid?
+Use the `TIMESTAMP` type instead of `DATE` whenever possible. Presently, the `DATE` type is serialized/deserialized very inefficiently resulting in performance degradation.
diff --git a/docs/_docs/code-snippets/dotnet/PersistenceTuning.cs b/docs/_docs/code-snippets/dotnet/PersistenceTuning.cs
new file mode 100644
index 0000000..448b577
--- /dev/null
+++ b/docs/_docs/code-snippets/dotnet/PersistenceTuning.cs
@@ -0,0 +1,79 @@
+using Apache.Ignite.Core;
+using Apache.Ignite.Core.Configuration;
+
+namespace dotnet_helloworld
+{
+ public class PersistenceTuning
+ {
+ public static void AdjustingPageSize()
+ {
+ // tag::page-size[]
+ var cfg = new IgniteConfiguration
+ {
+ DataStorageConfiguration = new DataStorageConfiguration
+ {
+ // Changing the page size to 4 KB.
+ PageSize = 4096
+ }
+ };
+ // end::page-size[]
+ }
+
+ public static void KeepWalsSeparately()
+ {
+ // tag::separate-wal[]
+ var cfg = new IgniteConfiguration
+ {
+ DataStorageConfiguration = new DataStorageConfiguration
+ {
+ // Sets a path to the root directory where data and indexes are to be persisted.
+ // It's assumed the directory is on a separated SSD.
+ StoragePath = "/ssd/storage",
+
+ // Sets a path to the directory where WAL is stored.
+ // It's assumed the directory is on a separated HDD.
+ WalPath = "/wal",
+
+ // Sets a path to the directory where WAL archive is stored.
+ // The directory is on the same HDD as the WAL.
+ WalArchivePath = "/wal/archive"
+ }
+ };
+ // end::separate-wal[]
+ }
+
+ public static void Throttling()
+ {
+ // tag::throttling[]
+ var cfg = new IgniteConfiguration
+ {
+ DataStorageConfiguration = new DataStorageConfiguration
+ {
+ WriteThrottlingEnabled = true
+ }
+ };
+ // end::throttling[]
+ }
+
+ public static void CheckpointBufferSize()
+ {
+ // tag::checkpointing-buffer-size[]
+ var cfg = new IgniteConfiguration
+ {
+ DataStorageConfiguration = new DataStorageConfiguration
+ {
+ WriteThrottlingEnabled = true,
+ DefaultDataRegionConfiguration = new DataRegionConfiguration
+ {
+ Name = DataStorageConfiguration.DefaultDataRegionName,
+ PersistenceEnabled = true,
+
+ // Increasing the buffer size to 1 GB.
+ CheckpointPageBufferSize = 1024L * 1024 * 1024
+ }
+ }
+ };
+ // end::checkpointing-buffer-size[]
+ }
+ }
+}
\ No newline at end of file
diff --git a/docs/_docs/code-snippets/java/pom.xml b/docs/_docs/code-snippets/java/pom.xml
index e0bcbe9..5f65802 100644
--- a/docs/_docs/code-snippets/java/pom.xml
+++ b/docs/_docs/code-snippets/java/pom.xml
@@ -66,6 +66,12 @@
<artifactId>ignite-aws</artifactId>
<version>${ignite.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.ignite</groupId>
+ <artifactId>ignite-compress</artifactId>
+ <version>${ignite.version}</version>
+ </dependency>
+
<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-gce</artifactId>
diff --git a/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/DiskCompression.java b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/DiskCompression.java
new file mode 100644
index 0000000..f2942b1
--- /dev/null
+++ b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/DiskCompression.java
@@ -0,0 +1,41 @@
+package org.apache.ignite.snippets;
+
+import org.apache.ignite.Ignite;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.DiskPageCompression;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.junit.jupiter.api.Test;
+
+public class DiskCompression {
+
+ @Test
+ void configuration() {
+ //tag::configuration[]
+ DataStorageConfiguration dsCfg = new DataStorageConfiguration();
+
+ //set the page size to 2 types of the disk page size
+ dsCfg.setPageSize(4096 * 2);
+
+ //enable persistence for the default data region
+ dsCfg.setDefaultDataRegionConfiguration(new DataRegionConfiguration().setPersistenceEnabled(true));
+
+ IgniteConfiguration cfg = new IgniteConfiguration();
+ cfg.setDataStorageConfiguration(dsCfg);
+
+ CacheConfiguration cacheCfg = new CacheConfiguration("myCache");
+ //enable disk page compression for this cache
+ cacheCfg.setDiskPageCompression(DiskPageCompression.LZ4);
+ //optionally set the compression level
+ cacheCfg.setDiskPageCompressionLevel(10);
+
+ cfg.setCacheConfiguration(cacheCfg);
+
+ Ignite ignite = Ignition.start(cfg);
+ //end::configuration[]
+
+ ignite.close();
+ }
+}
diff --git a/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/IgnitePersistence.java b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/IgnitePersistence.java
index 51a95cad..cf05707 100644
--- a/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/IgnitePersistence.java
+++ b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/IgnitePersistence.java
@@ -3,11 +3,14 @@ package org.apache.ignite.snippets;
import org.apache.ignite.Ignite;
import org.apache.ignite.Ignition;
import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.DiskPageCompression;
import org.apache.ignite.configuration.IgniteConfiguration;
+import org.junit.jupiter.api.Test;
public class IgnitePersistence {
- public static void disablingWal() {
+ @Test
+ void disablingWal() {
//tag::wal[]
IgniteConfiguration cfg = new IgniteConfiguration();
@@ -17,11 +20,11 @@ public class IgnitePersistence {
cfg.setDataStorageConfiguration(storageCfg);
Ignite ignite = Ignition.start(cfg);
-
+
ignite.cluster().active(true);
-
+
String cacheName = "myCache";
-
+
ignite.getOrCreateCache(cacheName);
ignite.cluster().disableWal(cacheName);
@@ -32,9 +35,10 @@ public class IgnitePersistence {
//end::wal[]
ignite.close();
}
-
- public static void changeWalSegmentSize() {
- // tag::segment-size[]
+
+ @Test
+ public static void changeWalSegmentSize() {
+ // tag::segment-size[]
IgniteConfiguration cfg = new IgniteConfiguration();
DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
@@ -44,11 +48,12 @@ public class IgnitePersistence {
cfg.setDataStorageConfiguration(storageCfg);
Ignite ignite = Ignition.start(cfg);
- // end::segment-size[]
-
+ // end::segment-size[]
+
ignite.close();
}
+ @Test
public static void cfgExample() {
//tag::cfg[]
IgniteConfiguration cfg = new IgniteConfiguration();
@@ -57,7 +62,7 @@ public class IgnitePersistence {
DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
-
+
//tag::storage-path[]
storageCfg.setStoragePath("/opt/storage");
//end::storage-path[]
@@ -68,13 +73,24 @@ public class IgnitePersistence {
//end::cfg[]
ignite.close();
}
-
- public static void main(String[] args) {
- IgnitePersistence ip = new IgnitePersistence();
-
- ip.disablingWal();
- ip.changeWalSegmentSize();
- ip.cfgExample();
+
+ @Test
+ void walRecordsCompression() {
+ //tag::wal-records-compression[]
+ IgniteConfiguration cfg = new IgniteConfiguration();
+
+ DataStorageConfiguration dsCfg = new DataStorageConfiguration();
+ dsCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
+
+ //WAL page compression parameters
+ dsCfg.setWalPageCompression(DiskPageCompression.LZ4);
+ dsCfg.setWalPageCompressionLevel(8);
+
+ cfg.setDataStorageConfiguration(dsCfg);
+ Ignite ignite = Ignition.start(cfg);
+ //end::wal-records-compression[]
+
+ ignite.close();
}
}
diff --git a/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/PersistenceTuning.java b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/PersistenceTuning.java
new file mode 100644
index 0000000..fb23a80
--- /dev/null
+++ b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/PersistenceTuning.java
@@ -0,0 +1,93 @@
+package org.apache.ignite.snippets;
+
+import org.apache.ignite.Ignite;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+
+public class PersistenceTuning {
+
+ void pageSize() {
+
+ // tag::page-size[]
+ IgniteConfiguration cfg = new IgniteConfiguration();
+
+ // Durable memory configuration.
+ DataStorageConfiguration storageCfg = new DataStorageConfiguration();
+
+ // Changing the page size to 8 KB.
+ storageCfg.setPageSize(8192);
+
+ cfg.setDataStorageConfiguration(storageCfg);
+ // end::page-size[]
+ }
+
+ void separateWal() {
+ // tag::separate-wal[]
+ IgniteConfiguration cfg = new IgniteConfiguration();
+
+ // Configuring Native Persistence.
+ DataStorageConfiguration storeCfg = new DataStorageConfiguration();
+
+ // Sets a path to the root directory where data and indexes are to be persisted.
+ // It's assumed the directory is on a separated SSD.
+ storeCfg.setStoragePath("/ssd/storage");
+
+ // Sets a path to the directory where WAL is stored.
+ // It's assumed the directory is on a separated HDD.
+ storeCfg.setWalPath("/wal");
+
+ // Sets a path to the directory where WAL archive is stored.
+ // The directory is on the same HDD as the WAL.
+ storeCfg.setWalArchivePath("/wal/archive");
+
+ cfg.setDataStorageConfiguration(storeCfg);
+
+ // Starting the node.
+ Ignite ignite = Ignition.start(cfg);
+
+ // end::separate-wal[]
+
+ ignite.close();
+ }
+
+ void writesThrottling() {
+ // tag::throttling[]
+ IgniteConfiguration cfg = new IgniteConfiguration();
+
+ // Configuring Native Persistence.
+ DataStorageConfiguration storeCfg = new DataStorageConfiguration();
+
+ // Enabling the writes throttling.
+ storeCfg.setWriteThrottlingEnabled(true);
+
+ cfg.setDataStorageConfiguration(storeCfg);
+ // Starting the node.
+ Ignite ignite = Ignition.start(cfg);
+ // end::throttling[]
+
+ ignite.close();
+ }
+
+ void checkpointingBufferSize() {
+ // tag::checkpointing-buffer-size[]
+ IgniteConfiguration cfg = new IgniteConfiguration();
+
+ // Configuring Native Persistence.
+ DataStorageConfiguration storeCfg = new DataStorageConfiguration();
+
+ // Enabling the writes throttling.
+ storeCfg.setWriteThrottlingEnabled(true);
+
+ // Increasing the buffer size to 1 GB.
+ storeCfg.getDefaultDataRegionConfiguration().setCheckpointPageBufferSize(1024L * 1024 * 1024);
+
+ cfg.setDataStorageConfiguration(storeCfg);
+
+ // Starting the node.
+ Ignite ignite = Ignition.start(cfg);
+ // end::checkpointing-buffer-size[]
+ ignite.close();
+ }
+
+}
diff --git a/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/WAL.java b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/WAL.java
new file mode 100644
index 0000000..45a33ad
--- /dev/null
+++ b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/WAL.java
@@ -0,0 +1,30 @@
+package org.apache.ignite.snippets;
+
+import org.apache.ignite.Ignite;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.DiskPageCompression;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.junit.jupiter.api.Test;
+
+public class WAL {
+
+ @Test
+ void walRecordsCompression() {
+ //tag::records-compression[]
+ IgniteConfiguration cfg = new IgniteConfiguration();
+
+ DataStorageConfiguration dsCfg = new DataStorageConfiguration();
+ dsCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
+
+ //WAL page compression parameters
+ dsCfg.setWalPageCompression(DiskPageCompression.LZ4);
+ dsCfg.setWalPageCompressionLevel(8);
+
+ cfg.setDataStorageConfiguration(dsCfg);
+ Ignite ignite = Ignition.start(cfg);
+ //end::records-compression[]
+
+ ignite.close();
+ }
+}
diff --git a/docs/_docs/code-snippets/xml/disk-compression.xml b/docs/_docs/code-snippets/xml/disk-compression.xml
new file mode 100644
index 0000000..53eb1d7
--- /dev/null
+++ b/docs/_docs/code-snippets/xml/disk-compression.xml
@@ -0,0 +1,43 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<beans xmlns="http://www.springframework.org/schema/beans" xmlns:util="http://www.springframework.org/schema/util" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">
+ <!-- tag::ignite-config[] -->
+ <bean class="org.apache.ignite.configuration.IgniteConfiguration">
+ <property name="dataStorageConfiguration">
+ <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
+ <property name="pageSize" value="#{4096 * 2}"/>
+ <property name="defaultDataRegionConfiguration">
+ <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
+ <property name="persistenceEnabled" value="true"/>
+ </bean>
+ </property>
+ </bean>
+ </property>
+ <property name="cacheConfiguration">
+ <bean class="org.apache.ignite.configuration.CacheConfiguration">
+ <property name="name" value="myCache"/>
+ <!-- enable disk page compression for this cache -->
+ <property name="diskPageCompression" value="LZ4"/>
+ <!-- optionally set the compression level -->
+ <property name="diskPageCompressionLevel" value="10"/>
+ </bean>
+ </property>
+ <!-- tag::discovery[] -->
+ <property name="discoverySpi">
+ <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
+ <!-- prevent this client from reconnecting on connection loss -->
+ <property name="clientReconnectDisabled" value="true"/>
+ <property name="ipFinder">
+ <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
+ <property name="addresses">
+ <list>
+ <value>127.0.0.1:47500..47509</value>
+ </list>
+ </property>
+ </bean>
+ </property>
+ </bean>
+ </property>
+ <!-- end::discovery[] -->
+ </bean>
+ <!-- end::ignite-config[] -->
+</beans>
\ No newline at end of file
diff --git a/docs/_docs/code-snippets/xml/persistence-tuning.xml b/docs/_docs/code-snippets/xml/persistence-tuning.xml
new file mode 100644
index 0000000..46d21e3
--- /dev/null
+++ b/docs/_docs/code-snippets/xml/persistence-tuning.xml
@@ -0,0 +1,65 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<beans xmlns="http://www.springframework.org/schema/beans" xmlns:util="http://www.springframework.org/schema/util" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">
+ <!-- tag::ignite-config[] -->
+ <bean class="org.apache.ignite.configuration.IgniteConfiguration">
+ <!-- tag::ds[] -->
+ <property name="dataStorageConfiguration">
+ <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
+
+ <!-- tag::page-size[] -->
+ <!-- Set the page size to 8 KB -->
+ <property name="pageSize" value="#{8 * 1024}"/>
+ <!-- end::page-size[] -->
+ <!-- tag::paths[] -->
+ <!--
+ Sets a path to the root directory where data and indexes are
+ to be persisted. It's assumed the directory is on a separated SSD.
+ -->
+ <property name="storagePath" value="/opt/persistence"/>
+ <property name="walPath" value="/opt/wal"/>
+ <property name="walArchivePath" value="/opt/wal-archive"/>
+ <!-- end::paths[] -->
+ <!-- tag::page-write-throttling[] -->
+ <property name="writeThrottlingEnabled" value="true"/>
+
+ <!-- end::page-write-throttling[] -->
+ <!-- tag::data-region[] -->
+ <property name="defaultDataRegionConfiguration">
+ <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
+ <!-- Enabling persistence. -->
+ <property name="persistenceEnabled" value="true"/>
+ <!-- Increasing the buffer size to 1 GB. -->
+ <property name="checkpointPageBufferSize" value="#{1024L * 1024 * 1024}"/>
+ </bean>
+ </property>
+
+ <!-- end::data-region[] -->
+ </bean>
+ </property>
+ <!-- end::ds[] -->
+ <!-- tag::discovery[] -->
+ <property name="discoverySpi">
+ <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
+ <property name="ipFinder">
+ <!--
+ Ignite provides several options for automatic discovery that can be used
+ instead os static IP based discovery. For information on all options refer
+ to our documentation: http://apacheignite.readme.io/docs/cluster-config
+ -->
+ <!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
+ <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
+ <!--bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder"-->
+ <property name="addresses">
+ <list>
+ <!-- In distributed environment, replace with actual host IP address. -->
+ <value>127.0.0.1:47500..47509</value>
+ </list>
+ </property>
+ </bean>
+ </property>
+ </bean>
+ </property>
+ <!-- end::discovery[] -->
+ </bean>
+ <!-- end::ignite-config[] -->
+</beans>
\ No newline at end of file
diff --git a/docs/_docs/configuring-caches/configuration-overview.adoc b/docs/_docs/configuring-caches/configuration-overview.adoc
index a55a705..e2b0277 100644
--- a/docs/_docs/configuring-caches/configuration-overview.adoc
+++ b/docs/_docs/configuring-caches/configuration-overview.adoc
@@ -61,7 +61,7 @@ a| This parameter controls the way the rebalancing process is performed. Possibl
| [[readfrombackup]] Read requested cache entry from the backup partition if it is available on the local node instead of making a request to the primary partition (which can be located on the remote nodes).
| `true`
-|`queryPrallelism` | The number of threads in a single node to process a SQL query executed on the cache. Refer to the link:perf-troubleshooting-guide/sql-tuning#query-parallelism[Query Parallelism] section in the Performance guide for more information.
+|`queryPrallelism` | The number of threads in a single node to process a SQL query executed on the cache. Refer to the link:SQL/sql-tuning#query-parallelism[Query Parallelism] section in the Performance guide for more information.
| 1
|===
diff --git a/docs/_docs/events/events.adoc b/docs/_docs/events/events.adoc
index 6c20477..aa7fccd 100644
--- a/docs/_docs/events/events.adoc
+++ b/docs/_docs/events/events.adoc
@@ -311,6 +311,7 @@ They allow you to get notification about different stages of transaction executi
|===
+////
== Management Task Events
Management task events represent the tasks that are executed by Visor or Web Console.
@@ -323,3 +324,4 @@ This event type can be used to monitor a link:security/cluster-monitor-audit[Web
|===
+////
diff --git a/docs/_docs/images/set-streaming.png b/docs/_docs/images/set-streaming.png
new file mode 100644
index 0000000..a448b59
Binary files /dev/null and b/docs/_docs/images/set-streaming.png differ
diff --git a/docs/_docs/installation/kubernetes/generic-configuration.adoc b/docs/_docs/installation/kubernetes/generic-configuration.adoc
index 7c440f8..8230953 100644
--- a/docs/_docs/installation/kubernetes/generic-configuration.adoc
+++ b/docs/_docs/installation/kubernetes/generic-configuration.adoc
@@ -159,7 +159,7 @@ Our Deployment configuration will deploy a ReplicaSet with two pods running Igni
In the container's configuration, we will:
-* Enable the “ignite-kubernetes” and “ignite-rest-http” link:installing-using-docker#enabling-modules[modules].
+* Enable the “ignite-kubernetes” and “ignite-rest-http” link:installation/installing-using-docker#enabling-modules[modules].
* Use the configuration file from the ConfigMap we created earlier.
* Open a number of ports:
** 47100 — the communication port
diff --git a/docs/_docs/monitoring-metrics/intro.adoc b/docs/_docs/monitoring-metrics/intro.adoc
index 760b433..5b5c5b7 100644
--- a/docs/_docs/monitoring-metrics/intro.adoc
+++ b/docs/_docs/monitoring-metrics/intro.adoc
@@ -26,7 +26,7 @@ Note that a node consists of several layers: hardware, the operating system, the
* For log based monitoring, the key is that you can act proactively, watch the logs for trends/etc., don't just wait to check the logs until something breaks.
- Network: ping monitoring, network hardware monitoring, TCP dumps
-This should give you a good place to start for setting up monitoring of your hardware, operating system, and network. To monitor the application layer (the nodes that make up your in-memory computing solution), you'll need to perform Ignite-specific monitoring via metrics you access with JMX/Beans or Web Console, or programmatically.
+This should give you a good place to start for setting up monitoring of your hardware, operating system, and network. To monitor the application layer (the nodes that make up your in-memory computing solution), you'll need to perform Ignite-specific monitoring via metrics you access with JMX/Beans or programmatically.
== Global vs. Node-specific Metrics
diff --git a/docs/_docs/monitoring-metrics/metrics.adoc b/docs/_docs/monitoring-metrics/metrics.adoc
index a8db872..f50c30a 100644
--- a/docs/_docs/monitoring-metrics/metrics.adoc
+++ b/docs/_docs/monitoring-metrics/metrics.adoc
@@ -177,7 +177,7 @@ Checkpointing may slow down cluster operations.
You may want to monitor how much time each checkpoint operation takes, so that you can tune the properties that affect checkpointing.
You may also want to monitor the disk performance to see if the slow-down is caused by external reasons.
-See link:perf-troubleshooting-guide/persistence-tuning#pages-writes-throttling[Pages Writes Throttling] and link:perf-troubleshooting-guide/persistence-tuning#adjusting-checkpointing-buffer-size[Checkpointing Buffer Size] for performance tips.
+See link:persistence/persistence-tuning#pages-writes-throttling[Pages Writes Throttling] and link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[Checkpointing Buffer Size] for performance tips.
Mbean's Object Name: ::
+
diff --git a/docs/_docs/monitoring-metrics/system-views.adoc b/docs/_docs/monitoring-metrics/system-views.adoc
index 41f1df4..3f61381 100644
--- a/docs/_docs/monitoring-metrics/system-views.adoc
+++ b/docs/_docs/monitoring-metrics/system-views.adoc
@@ -258,9 +258,6 @@ This view contains information about the SQL queries currently executing on the
| DURATION| BIGINT | The duration of the query up to the current moment.
| MEMORY_CURRENT| BIGINT | The amount of memory the query uses at the current moment, in bytes.
| MEMORY_MAX| BIGINT | The maximum amount of memory the query has used during its execution, in bytes.
-| DISK_ALLOCATION_CURRENT| BIGINT | The amount of disk space the query uses at the moment, in bytes. See link:memory-configuration/memory-quotas#query-offloading[Query Offloading] for details.
-| DISK_ALLOCATION_MAX| BIGINT | Maximum amount of disk space the query has used during its execution, in bytes.
-| DISK_ALLOCATION_TOTAL| BIGINT | The query can allocate different amount of disk space at different stages of execution (because query execution follows the map-reduce pattern). This column returns the sum of those amounts, in bytes.
| INITIATOR_ID| VARCHAR a| A string identifying the entity that started the query. By default, the `initiator_id` has different format depending on where the query was started:
* link:SQL/JDBC/jdbc-driver[JDBC thin driver]: `jdbc-thin:<client_IP_host>:<client_IP_port>@<user_name>`
@@ -268,7 +265,7 @@ This view contains information about the SQL queries currently executing on the
* link:SQL/JDBC/jdbc-client-driver[JDBC client driver]: `jdbc-v2:<client_IP_host>:sqlGrid-ignite-jdbc-driver-<UUID>`
* link:distributed-computing/distributed-computing#executing-tasks[Task]: `<job_class_name>:<job_ID>`
-The query initiator can be set via SQL API: link:{javadoc_base_url}/org/apache/ignite/cache/query/SqlFieldsQuery.html[SqlFieldsQuery].
+The query initiator can be set via SQL API: javadoc:org.apache.ignite.cache.query.SqlFieldsQuery[].
|===
@@ -291,10 +288,6 @@ This system view contains the list of SQL queries executed on the node where the
| LAST_START_TIME| TIMESTAMP| The timestamp when the query was started last time.
| MEMORY_MIN| BIGINT| Minimum amount of memory the query has used, in bytes.
| MEMORY_MAX| BIGINT| Maximum amount of memory the query has used, in bytes.
-| DISK_ALLOCATION_MIN| BIGINT| Minimum allocated disk space the query has used. See link:memory-configuration/memory-quotas#query-offloading[Query Offloading] for details.
-| DISK_ALLOCATION_MAX| BIGINT| Maximum allocated disk space the query has used.
-| DISK_ALLOCATION_TOTAL_MIN| BIGINT| Minimum total allocated disk space the query has used. See the explanation for the `DISK_ALLOCATION_TOTAL` column in <<LOCAL_SQL_RUNNING_QUERIES>>.
-| DISK_ALLOCATION_TOTAL_MAX| BIGINT |Maximum total allocated disk space the query has used.
|===
@@ -318,14 +311,14 @@ The INDEXES view contains information about SQL indexes.
| CACHE_GROUP_NAME |VARCHAR | The cache group name.
| CACHE_NAME |VARCHAR | The cache name.
| CACHE_ID | INT | Cache ID.
-|INLINE_SIZE | INT | The link:perf-troubleshooting-guide/sql-tuning#increasing-index-inline-size[inline size] in bytes.
+|INLINE_SIZE | INT | The link:SQL/sql-tuning#increasing-index-inline-size[inline size] in bytes.
| IS_PK | BOOLEAN | Indicates whether the index is for the primary key.
| IS_UNIQUE | BOOLEAN | Indicates if the index is unique.
|===
== Examples
-To query the system views using the link:tools-analytics/sqlline[SQLLine] tool, connect to the SYS schema as follows:
+To query the system views using the link:sqlline[SQLLine] tool, connect to the SYS schema as follows:
[source, shell]
----
diff --git a/docs/_docs/persistence/disk-compression.adoc b/docs/_docs/persistence/disk-compression.adoc
new file mode 100644
index 0000000..09aad1f
--- /dev/null
+++ b/docs/_docs/persistence/disk-compression.adoc
@@ -0,0 +1,48 @@
+= Disk Compression
+
+Disk compression refers to the process of compressing data pages when they are written to disk, reducing the size of the on-disk storage.
+The pages are kept in memory uncompressed, but when the data is flushed to disk it is compressed using the configured algorithm.
+This applies only to data pages that are stored to the persistent storage and does not compress indexes or WAL records.
+link:[WAL records compression] can be enabled separately.
+
+Disk page compression can be enabled on a per cache basis in the cache configuration.
+The cache must reside in a persistent link:[data region].
+There is no option to enable disk page compression globally at the moment.
+Moreover, the following prerequisites must be met:
+
+* Set the `pageSize` property in your data storage configuration to at least 2 times the page size of your file system. It means that the page size must be either 8K or 16K.
+* Enable the `ignite-compress` module.
+
+To enable disk page compression for a cache, provide one of the available compression algorithms in the cache configuration, as shown in the following example:
+
+
+[tabs]
+--
+tab:XML[]
+
+[source, xml]
+----
+include::code-snippets/xml/disk-compression.xml[tags=ignite-config;!discovery, indent=0]
+----
+
+tab:Java[]
+
+[source, java]
+----
+include::{javaCodeDir}/DiskCompression.java[tags=configuration, indent=0]
+----
+
+tab:C#/.NET[]
+
+tab:C++[unsupported]
+
+--
+
+== Supported Algorithms
+
+The supported compression algorithms include:
+
+* `ZSTD` — supports compression levels from -131072 to 22 (default: 3).
+* `LZ4` — supports compression levels from 0 to 17 (default: 0).
+* `SNAPPY` — the Snappy algorithm.
+* `SKIP_GARBAGE` — this algorithm only extracts useful data from half-filled pages and does not compress the data.
diff --git a/docs/_docs/persistence/native-persistence.adoc b/docs/_docs/persistence/native-persistence.adoc
index a28dfd4..3381ad5 100644
--- a/docs/_docs/persistence/native-persistence.adoc
+++ b/docs/_docs/persistence/native-persistence.adoc
@@ -192,7 +192,7 @@ If a node is terminated abruptly during update operations, it is very likely tha
=== WAL Archive
The WAL archive is used to store WAL segments that may be needed to recover the node after a crash. The number of segments kept in the archive is such that the total size of all segments does not exceed the specified size of the WAL archive.
-By default, the maximum size of the WAL archive (total space it occupies on disk) is defined as 4 times the size of the link:perf-troubleshooting-guide/persistence-tuning#adjusting-checkpointing-buffer-size[checkpointing buffer]. You can change that value in the <<Configuration Properties,configuration>>.
+By default, the maximum size of the WAL archive (total space it occupies on disk) is defined as 4 times the size of the link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[checkpointing buffer]. You can change that value in the <<Configuration Properties,configuration>>.
CAUTION: Setting the WAL archive size to a value lower than the default may impact performance and should be tested before being used in production.
@@ -265,6 +265,32 @@ If the segments are needed (for example, to re-balance data between nodes), they
See the <<Configuration Properties>> section below to learn how to enable WAL archive compaction.
+=== WAL Records Compression
+
+As described in the link:https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-WAL[design document], physical and logical records that represent data updates are written to the WAL files before the user operation is acknowledged.
+Ignite can compress WAL records in memory before they are written to disk to save space.
+
+WAL Records Compression requires that the 'ignite-compress' module be enabled. See link:setup#enabling-modules[Enabling Modules].
+
+By default WAL records compression is disabled.
+To enable it, set the compression algorithm and compression level in the data storage configuration:
+
+[tabs]
+--
+tab:XML[]
+
+tab:Java[]
+[source, java]
+----
+include::{javaFile}[tags=wal-records-compression, indent=0]
+----
+
+tab:C#/.NET[]
+tab:C++[unsupported]
+--
+
+The supported compression algorithms are listed in javadoc:org.apache.ignite.configuration.DiskPageCompression[].
+
=== Disabling WAL Archive
In some cases, you may want to disable WAL archiving, for example, to reduce the overhead associated with copying of WAL segments to the archive. There can be a situation where Ignite writes data to WAL segments faster than the segments are copied to the archive. This may create an I/O bottleneck that can freeze the operation of the node. If you experience such problems, try disabling WAL archiving.
@@ -297,7 +323,7 @@ This process helps to utilize disk space frugally by keeping pages in the most u
See the following related documentation:
* link:monitoring-metrics/metrics#monitoring-checkpointing-operations[Monitoring Checkpointing Operations].
-* link:perf-troubleshooting-guide/persistence-tuning#adjusting-checkpointing-buffer-size[Adjusting Checkpointing Buffer Size]
+* link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[Adjusting Checkpointing Buffer Size]
== Configuration Properties
@@ -318,5 +344,5 @@ The following table describes some properties of link:{javadoc_base_url}/org/apa
|`walMode` | <<WAL Modes,Write-ahead logging mode>>. | `LOG_ONLY`
| `walCompactionLevel` | WAL archive compression level. `1` indicates the fastest speed, and `9` indicates the best compression. | `1`
-|`maxWalArchiveSize` | The maximum size (in bytes) the WAL archive can occupy on the file system. | Four times the size of the link:perf-troubleshooting-guide/persistence-tuning#adjusting-checkpointing-buffer-size[checkpointing buffer].
+|`maxWalArchiveSize` | The maximum size (in bytes) the WAL archive can occupy on the file system. | Four times the size of the link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[checkpointing buffer].
|=======================================================================
diff --git a/docs/_docs/persistence/persistence-tuning.adoc b/docs/_docs/persistence/persistence-tuning.adoc
new file mode 100644
index 0000000..e7324c6
--- /dev/null
+++ b/docs/_docs/persistence/persistence-tuning.adoc
@@ -0,0 +1,244 @@
+= Persistence Tuning
+:javaFile: {javaCodeDir}/PersistenceTuning.java
+:xmlFile: code-snippets/xml/persistence-tuning.xml
+:dotnetFile: code-snippets/dotnet/PersistenceTuning.cs
+
+This article summarizes best practices for Ignite native persistence tuning.
+If you are using an external (3rd party) storage for persistence needs, please refer to performance guides from the 3rd party vendor.
+
+== Adjusting Page Size
+
+The `DataStorageConfiguration.pageSize` parameter should be no less than the lower of: the page size of your storage media (SSD, Flash, HDD, etc.) and the cache page size of your operating system.
+The default value is 4KB.
+
+The operating system's cache page size can be easily checked using
+link:https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space[system tools and parameters, window=_blank].
+
+The page size of the storage device such as SSD is usually noted in the device specification. If the manufacturer does not disclose this information, try to run SSD benchmarks to figure out the number.
+Many manufacturers have to adapt their drivers for 4 KB random-write workloads because a variety of standard
+benchmarks use 4 KB by default.
+link:https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf[This white paper,window=_blank] from Intel confirms that 4 KB should be enough.
+
+Once you pick the most optimal page size, apply it in your cluster configuration:
+
+////
+TODO for .NET and other languages.
+////
+
+[tabs]
+--
+tab:XML[]
+[source,xml]
+----
+include::{xmlFile}[tags=!*;ignite-config;ds;page-size,indent=0]
+----
+tab:Java[]
+[source,java]
+----
+include::{javaFile}[tag=page-size,indent=0]
+----
+tab:C#/.NET[]
+[source,csharp]
+----
+include::{dotnetFile}[tag=page-size,indent=0]
+----
+tab:C++[unsupported]
+--
+
+== Keep WALs Separately
+
+Consider using separate drives for data files and link:persistence/native-persistence#write-ahead-log[Write-Ahead-Logging (WAL)].
+Ignite actively writes to both the data and WAL files.
+
+The example below shows how to configure separate paths for the data storage, WAL, and WAL archive:
+
+[tabs]
+--
+tab:XML[]
+[source,xml]
+----
+include::{xmlFile}[tags=!*;ignite-config;ds;paths,indent=0]
+----
+tab:Java[]
+[source,java]
+----
+include::{javaFile}[tag=separate-wal,indent=0]
+----
+tab:C#/.NET[]
+[source,csharp]
+----
+include::{dotnetFile}[tag=separate-wal,indent=0]
+----
+tab:C++[unsupported]
+--
+
+== Increasing WAL Segment Size
+
+The default WAL segment size (64 MB) may be inefficient in high load scenarios because it causes WAL to switch between segments too frequently and switching/rotation is a costly operation. Setting the segment size to a higher value (up to 2 GB) may help reduce the number of switching operations. However, the tradeoff is that this will increase the overall volume of the write-ahead log.
+
+See link:persistence/native-persistence#changing-wal-segment-size[Changing WAL Segment Size] for details.
+
+== Changing WAL Mode
+
+Consider other WAL modes as alternatives to the default mode. Each mode provides different degrees of reliability in
+case of node failure and that degree is inversely proportional to speed, i.e. the more reliable the WAL mode, the
+slower it is. Therefore, if your use case does not require high reliability, you can switch to a less reliable mode.
+
+See link:persistence/native-persistence#wal-modes[WAL Modes] for more details.
+
+== Disabling WAL
+
+//TODO: when should bhis be done?
+There are situations where link:persistence/native-persistence#disabling-wal[disabling the WAL] can help improve performance.
+
+== Pages Writes Throttling
+
+Ignite periodically starts the link:persistence/native-persistence#checkpointing[checkpointing process] that syncs dirty pages from memory to disk. A dirty page is a page that was updated in RAM but was not written to a respective partition file (an update was just appended to the WAL). This process happens in the background without affecting the application's logic.
+
+However, if a dirty page, scheduled for checkpointing, is updated before being written to disk, its previous state is copied to a special region called a checkpointing buffer.
+If the buffer gets overflowed, Ignite will stop processing all updates until the checkpointing is over.
+As a result, write performance can drop to zero as shown in this diagram, until the checkpointing cycle is completed:
+
+image::images/checkpointing-chainsaw.png[Checkpointing Chainsaw]
+
+The same situation occurs if the dirty pages threshold is reached again while the checkpointing is in progress.
+This will force Ignite to schedule one more checkpointing execution and to halt all the update operations until the first checkpointing cycle is over.
+
+Both situations usually arise when either a disk device is slow or the update rate is too intensive.
+To mitigate and prevent these performance drops, consider enabling the pages write throttling algorithm.
+The algorithm brings the performance of update operations down to the speed of the disk device whenever the checkpointing buffer fills in too fast or the percentage of dirty pages soar rapidly.
+
+[NOTE]
+====
+[discrete]
+=== Pages Write Throttling in a Nutshell
+
+Refer to the link:https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-PagesWriteThrottling[Ignite wiki page, window=_blank] maintained by Apache Ignite persistence experts to get more details about throttling and its causes.
+====
+
+The example below shows how to enable write throttling:
+
+[tabs]
+--
+tab:XML[]
+[source,xml]
+----
+include::{xmlFile}[tags=!*;ignite-config;ds;page-write-throttling,indent=0]
+----
+tab:Java[]
+[source,java]
+----
+include::{javaFile}[tag=throttling,indent=0]
+----
+tab:C#/.NET[]
+[source,csharp]
+----
+include::{dotnetFile}[tag=throttling,indent=0]
+----
+tab:C++[unsupported]
+--
+
+== Adjusting Checkpointing Buffer Size
+
+The size of the checkpointing buffer, explained in the previous section, is one of the checkpointing process triggers.
+
+The default buffer size is calculated as a function of the link:memory-configuration/data-regions[data region] size:
+
+[width=100%,cols="1,2",options="header"]
+|=======================================================================
+| Data Region Size |Default Checkpointing Buffer Size
+
+|< 1 GB | MIN (256 MB, Data_Region_Size)
+
+|between 1 GB and 8 GB | Data_Region_Size / 4
+
+|> 8 GB | 2 GB
+
+|=======================================================================
+
+The default buffer size can be suboptimal for write-intensive workloads because the page write throttling algorithm will slow down your writes whenever the size reaches the critical mark.
+To keep write performance at the desired pace while the checkpointing is in progress, consider increasing
+`DataRegionConfiguration.checkpointPageBufferSize` and enabling write throttling to prevent performance drops:
+
+[tabs]
+--
+tab:XML[]
+[source,xml]
+----
+include::{xmlFile}[tags=!*;ignite-config;ds;page-write-throttling;data-region,indent=0]
+----
+tab:Java[]
+[source,java]
+----
+include::{javaFile}[tag=checkpointing-buffer-size,indent=0]
+----
+tab:C#/.NET[]
+[source,csharp]
+----
+include::{dotnetFile}[tag=checkpointing-buffer-size,indent=0]
+----
+tab:C++[unsupported]
+--
+
+In the example above, the checkpointing buffer size of the default region is set to 1 GB.
+
+////
+TODO: describe when checkpointing is triggered
+[NOTE]
+====
+[discrete]
+=== When is the Checkpointing Process Triggered?
+
+Checkpointing is started if either the dirty pages count goes beyond the `totalPages * 2 / 3` value or
+`DataRegionConfiguration.checkpointPageBufferSize` is reached. However, if page write throttling is used, then
+`DataRegionConfiguration.checkpointPageBufferSize` is never encountered because it cannot be reached due to the way the algorithm works.
+====
+////
+
+== Enabling Direct I/O
+//TODO: why is this not enabled by default?
+Usually, whenever an application reads data from disk, the OS gets the data and puts it in a file buffer cache first.
+Similarly, for every write operation, the OS first writes the data in the cache and transfers it to disk later. To
+eliminate this process, you can enable Direct I/O in which case the data is read and written directly from/to the
+disk, bypassing the file buffer cache.
+
+The Direct I/O module in Ignite is used to speed up the checkpointing process, which writes dirty pages from RAM to disk. Consider using the Direct I/O plugin for write-intensive workloads.
+
+[NOTE]
+====
+[discrete]
+=== Direct I/O and WALs
+
+Note that Direct I/O cannot be enabled specifically for WAL files. However, enabling the Direct I/O module provides
+a slight benefit regarding the WAL files as well: the WAL data will not be stored in the OS buffer cache for too long;
+it will be flushed (depending on the WAL mode) at the next page cache scan and removed from the page cache.
+====
+
+You can enable Direct I/O, move the `{IGNITE_HOME}/libs/optional/ignite-direct-io` folder to the upper level `libs/optional/ignite-direct-io` folder in your Ignite distribution or as a Maven dependency as described link:setup#enabling-modules[here].
+
+You can use the `IGNITE_DIRECT_IO_ENABLED` system property to enable or disable the plugin at runtime.
+
+Get more details from the link:https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-DirectI/O[Ignite Direct I/O Wiki section, window=_blank].
+
+== Purchase Production-Level SSDs
+
+Note that the performance of Ignite Native Persistence may drop after several hours of intensive write load due to
+the nature of how link:http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-architecture-of-an-ssd-and-benchmarking[SSDs are designed and operate, window=_blank].
+Consider buying fast production-level SSDs to keep the performance high or switch to non-volatile memory devices like
+Intel Optane Persistent Memory.
+
+== SSD Over-provisioning
+
+Performance of random writes on a 50% filled disk is much better than on a 90% filled disk because of the SSDs over-provisioning (see link:https://www.seagate.com/tech-insights/ssd-over-provisioning-benefits-master-ti[https://www.seagate.com/tech-insights/ssd-over-provisioning-benefits-master-ti, window=_blank]).
+
+Consider buying SSDs with higher over-provisioning rates and make sure the manufacturer provides the tools to adjust it.
+
+[NOTE]
+====
+[discrete]
+=== Intel 3D XPoint
+
+Consider using 3D XPoint drives instead of regular SSDs to avoid the bottlenecks caused by a low over-provisioning
+setting and constant garbage collection at the SSD level.
+Read more link:http://dmagda.blogspot.com/2017/10/3d-xpoint-outperforms-ssds-verified-on.html[here, window=_blank].
+====
diff --git a/docs/_docs/sql-reference/ddl.adoc b/docs/_docs/sql-reference/ddl.adoc
index ca53cd8..6f35991 100644
--- a/docs/_docs/sql-reference/ddl.adoc
+++ b/docs/_docs/sql-reference/ddl.adoc
@@ -274,7 +274,7 @@ Parameters:
* `SPATIAL` - create the spatial index. Presently, only geometry types are supported.
* `IF NOT EXISTS` - do not throw an error if an index with the same name already exists. The database checks indexes' names only, and does not consider columns types or count.
* `index_option` - additional options for index creation:
-** `INLINE_SIZE` - specifies index inline size in bytes. Depending on the size, Ignite will place the whole indexed value or a part of it directly into index pages, thus omitting extra calls to data pages and increasing queries' performance. Index inlining is enabled by default and the size is pre-calculated automatically based on the table structure. To disable inlining, set the size to 0 (not recommended). Refer to the link:perf-troubleshooting-guide/sql-tuning#increasing-index-inline- [...]
+** `INLINE_SIZE` - specifies index inline size in bytes. Depending on the size, Ignite will place the whole indexed value or a part of it directly into index pages, thus omitting extra calls to data pages and increasing queries' performance. Index inlining is enabled by default and the size is pre-calculated automatically based on the table structure. To disable inlining, set the size to 0 (not recommended). Refer to the link:SQL/sql-tuning#increasing-index-inline-size[Increasing Index I [...]
** `PARALLEL` - specifies the number of threads to be used in parallel for index creation. The greater number is set, the faster the index is created and built. If the value exceeds the number of CPUs, then it will be decreased to the number of cores. If the parameter is not specified, then the number of threads is calculated as 25% of the CPU cores available.
diff --git a/docs/_docs/sql-reference/operational-commands.adoc b/docs/_docs/sql-reference/operational-commands.adoc
index 043ea2d..262a2b7 100644
--- a/docs/_docs/sql-reference/operational-commands.adoc
+++ b/docs/_docs/sql-reference/operational-commands.adoc
@@ -33,7 +33,7 @@ COPY FROM '/path/to/local/file.csv' INTO city (
ID, Name, CountryCode, District, Population) FORMAT CSV
----
-In the above command, substitute `/path/to/local/file.csv` with the actual path to your CSV file. For instance, you can use `city.csv` which is shipped with the latest Ignite.
+In the above command, substitute `/path/to/local/file.csv` with the actual path to your CSV file. For instance, you can use `city.csv` which is shipped with the latest Ignite.
You can find it in your `{IGNITE_HOME}/examples/src/main/resources/sql/` directory.
== SET STREAMING
@@ -101,7 +101,7 @@ While streaming mode allows you to load data much faster than other data loading
2. Due to streaming mode's asynchronous nature, you cannot know update counts for every statement executed; all JDBC/ODBC commands returning update counts will return 0.
=== Example
-As an example, you can use the sample world.sql file that is shipped with the latest Ignite distribution. It can be found in the `{IGNITE_HOME}/examples/sql/` directory. You can use the `run` command from link:tools-analytics/sqlline[SQLLine, window=_blank], as shown below:
+As an example, you can use the sample world.sql file that is shipped with the latest Ignite distribution. It can be found in the `{IGNITE_HOME}/examples/sql/` directory. You can use the `run` command from link:sqlline[SQLLine, window=_blank], as shown below:
[source,shell]
----
diff --git a/docs/_docs/sql-reference/sql-conformance.adoc b/docs/_docs/sql-reference/sql-conformance.adoc
index cabc8ff..ba0e9c4 100644
--- a/docs/_docs/sql-reference/sql-conformance.adoc
+++ b/docs/_docs/sql-reference/sql-conformance.adoc
@@ -280,9 +280,9 @@ Ignite does not support the following sub-features:
| `E171` SQLSTATE support
| Ignite provides partial support for this feature implementing a subset of standard error codes and introducing custom ones. A full list of errors supported by Ignite can be found here:
-link:developers-guide/SQL/JDBC/jdbc-driver#error-codes[JDBC Error Codes]
+link:SQL/JDBC/jdbc-driver#error-codes[JDBC Error Codes]
-link:developers-guide/SQL/ODBC/error-codes[ODBC Error Codes]
+link:SQL/ODBC/error-codes[ODBC Error Codes]
| `E182` Host language Binding (previously "Module Language")
| Ignite does not support this feature.
diff --git a/docs/_docs/sqlline.adoc b/docs/_docs/sqlline.adoc
new file mode 100644
index 0000000..20489b5
--- /dev/null
+++ b/docs/_docs/sqlline.adoc
@@ -0,0 +1,211 @@
+= SQLLine
+
+
+Command line tool for SQL connectivity.
+
+== Overview
+Apache Ignite is shipped with the SQLLine tool – a console-based utility for connecting to relational databases and executing SQL commands.
+This documentation describes how to connect SQLLine to your cluster, as well as various supported SQLLine commands.
+
+== Connecting to Ignite Cluster
+From your {IGNITE_HOME}/bin directory, run `sqlline.sh -u jdbc:ignite:thin:[host]` to connect SQLLine to the cluster. Substitute [host] with your actual value. For example:
+
+[tabs]
+--
+tab:Unix[]
+[source,shell]
+----
+./sqlline.sh --verbose=true -u jdbc:ignite:thin://127.0.0.1/
+----
+
+tab:Windows[]
+[source,shell]
+----
+sqlline.bat --verbose=true -u jdbc:ignite:thin://127.0.0.1/
+----
+
+--
+
+
+
+Use the `-h` or `help` option to see the various options available with SQLLine:
+
+[tabs]
+--
+tab:Unix[]
+[source,shell]
+----
+./sqlline.sh -h
+./sqlline.sh --help
+----
+
+tab:Windows[]
+[source,shell]
+----
+sqlline.bat -h
+sqlline.bat --help
+----
+--
+
+
+=== Authentication
+If you have authentication enabled for your cluster, then from your `{IGNITE_HOME}/bin' directory, run `jdbc:ignite:thin://[address]:[port];user=[username];password=[password]` to connect SQLLine to the cluster. Substitute `[address]`, `[port]`, `[username]` and `[password]` with your actual values. For example:
+
+
+[tabs]
+--
+tab:Unix[]
+[source,shell]
+----
+./sqlline.sh --verbose=true -u "jdbc:ignite:thin://127.0.0.1:10800;user=ignite;password=ignite"
+----
+
+tab:Windows[]
+[source,shell]
+----
+sqlline.bat --verbose=true -u "jdbc:ignite:thin://127.0.0.1:10800;user=ignite;password=ignite"
+----
+--
+
+If you do not have authentication set, omit `[username]` and `[password]`.
+
+[NOTE]
+====
+[discrete]
+=== Put JDBC URL in Quotes When Connecting from bash
+Make sure to put the connection URL in " " quotes when connecting from a bash environment, as follows: "jdbc:ignite:thin://[address]:[port];user=[username];password=[password]"
+====
+
+== Commands
+Here is the list of supported link:http://sqlline.sourceforge.net#commands[SQLLine commands, window=_blank]:
+
+[width="100%", cols="25%, 75%"]
+|=======
+|Command | Description
+
+|`!all`
+|Execute the specified SQL against all the current connections.
+
+|`!batch`
+|Start or execute a batch of SQL statements.
+
+|`!brief`
+|Enable terse output mode.
+
+|`!closeall`
+|Close all current open connections.
+
+|`!columns`
+|Display columns of a table.
+
+|`!connect`
+|Connect to a database.
+
+|`!dbinfo`
+|List metadata information about the current connection.
+
+|`!dropall`
+|Drop all tables in the database.
+
+|`!go`
+|Change to a different active connection.
+
+|`!help`
+|Display help information.
+
+|`!history`
+|Display the command history.
+
+|`!indexes`
+|Display indexes for a table.
+
+|`!list`
+|Display all active connections.
+
+|`!manual`
+|Display SQLLine manual.
+
+|`!metadata`
+|Invoke arbitrary metadata commands.
+
+|`!nickname`
+|Create a friendly name for the connection (updates command prompt).
+
+|`!outputformat`
+|Change the method for displaying SQL results.
+
+|`!primarykeys`
+|Display the primary key columns for a table.
+
+|`!properties`
+|Connect to the database defined in the specified properties file.
+
+|`!quit`
+|Exit SQLLine.
+
+|`!reconnect`
+|Reconnect to the current database.
+
+|`!record`
+|Begin recording all output from SQL commands.
+
+|`!run`
+|Execute a command script.
+
+|`!script`
+|Save executed commands to a file.
+
+|`!sql`
+|Execute a SQL against a database.
+
+|`!tables`
+|List all the tables in the database.
+
+|`!verbose`
+|Enable verbose output mode.
+|=======
+
+Note that the above list may not be complete. Support for additional SQLLine commands can be added.
+
+== Example
+After connecting to the cluster, you can execute SQL statements and SQLLine commands:
+
+
+Create tables:
+[source,sql]
+----
+0: jdbc:ignite:thin://127.0.0.1/> CREATE TABLE City (id LONG PRIMARY KEY, name VARCHAR) WITH "template=replicated";
+No rows affected (0.301 seconds)
+
+0: jdbc:ignite:thin://127.0.0.1/> CREATE TABLE Person (id LONG, name VARCHAR, city_id LONG, PRIMARY KEY (id, city_id))WITH "backups=1, affinityKey=city_id";
+No rows affected (0.078 seconds)
+
+0: jdbc:ignite:thin://127.0.0.1/> !tables
++-----------+--------------+--------------+-------------+-------------+
+| TABLE_CAT | TABLE_SCHEM | TABLE_NAME | TABLE_TYPE | REMARKS |
++-----------+--------------+--------------+-------------+-------------+
+| | PUBLIC | CITY | TABLE | |
+| | PUBLIC | PERSON | TABLE | |
++-----------+--------------+--------------+-------------+-------------+
+----
+
+Define indexes:
+
+[source,sql]
+----
+0: jdbc:ignite:thin://127.0.0.1/> CREATE INDEX idx_city_name ON City (name);
+No rows affected (0.039 seconds)
+
+0: jdbc:ignite:thin://127.0.0.1/> CREATE INDEX idx_person_name ON Person (name);
+No rows affected (0.013 seconds)
+
+0: jdbc:ignite:thin://127.0.0.1/> !indexes
++-----------+--------------+--------------+-------------+-----------------+
+| TABLE_CAT | TABLE_SCHEM | TABLE_NAME | NON_UNIQUE | INDEX_QUALIFIER |
++-----------+--------------+--------------+-------------+-----------------+
+| | PUBLIC | CITY | true | |
+| | PUBLIC | PERSON | true | |
++-----------+--------------+--------------+-------------+-----------------+
+----
+
+You can also watch a link:https://www.youtube.com/watch?v=FKS8A86h-VY[screencast, window=_blank] to learn more about how to use SQLLine.
diff --git a/docs/_docs/thin-clients/java-thin-client.adoc b/docs/_docs/thin-clients/java-thin-client.adoc
index 21ac06c..2a24446 100644
--- a/docs/_docs/thin-clients/java-thin-client.adoc
+++ b/docs/_docs/thin-clients/java-thin-client.adoc
@@ -5,8 +5,6 @@
The Java thin client is a lightweight client that connects to the cluster via a standard socket connection. It does not become a part of the cluster topology, never holds any data, and is not used as a destination for compute calculations. The thin client simply establishes a socket connection to a standard node and performs all operations through that node.
-To start a single node cluster that you can use to run examples, refer to the link:getting-started/quick-start/java[Java Quick Start Guide].
-
== Setting Up
If you use maven or gradle, add the `ignite-core` dependency to your application:
diff --git a/docs/_docs/thin-clients/nodejs-thin-client.adoc b/docs/_docs/thin-clients/nodejs-thin-client.adoc
index 2a540a0..4e244c4 100644
--- a/docs/_docs/thin-clients/nodejs-thin-client.adoc
+++ b/docs/_docs/thin-clients/nodejs-thin-client.adoc
@@ -187,7 +187,7 @@ include::{source_code_dir}/scanquery.js[tag="scan-query", indent=0]
----
== Executing SQL Statements
-The Node.js thin client supports all link:sql-reference/sql-reference-overview[SQL commands] that are supported by Ignite.
+The Node.js thin client supports all link:sql-reference[SQL commands] that are supported by Ignite.
The commands are executed via the `query(SqlFieldQuery)` method of the cache object.
The method accepts an instance of `SqlFieldsQuery` that represents a SQL statement and returns an instance of the `SqlFieldsCursor` class. Use the cursor to iterate over the result set or get all results at once.
diff --git a/docs/_docs/thin-clients/php-thin-client.adoc b/docs/_docs/thin-clients/php-thin-client.adoc
index c7b04d2..edb869a 100644
--- a/docs/_docs/thin-clients/php-thin-client.adoc
+++ b/docs/_docs/thin-clients/php-thin-client.adoc
@@ -102,7 +102,7 @@ include::code-snippets/php/UsingKeyValueApi.php[tag=scanQry,indent=0]
----
== Executing SQL Statements
-The PHP thin client supports all link:sql-reference/sql-reference-overview[SQL commands] that are supported by Ignite.
+The PHP thin client supports all link:sql-reference[SQL commands] that are supported by Ignite.
The commands are executed via the `query(SqlFieldQuery)` method of the cache object.
The method accepts an instance of `SqlFieldsQuery` that represents a SQL statement.
The `query()` method returns a cursor object with the standard PHP Iterator interface — use this cursor to iterate over the result set lazily, one by one. In addition, the cursor has methods to get all results at once.
diff --git a/docs/_docs/thin-clients/python-thin-client.adoc b/docs/_docs/thin-clients/python-thin-client.adoc
index 6fb9212..32fcb4f 100644
--- a/docs/_docs/thin-clients/python-thin-client.adoc
+++ b/docs/_docs/thin-clients/python-thin-client.adoc
@@ -378,11 +378,11 @@ NOTE: Be cautious: if the cache contains a large set of data, the dictionary may
== Executing SQL Statements
-The Python thin client supports all link:sql-reference/sql-reference-overview[SQL commands] that are supported by Ignite.
+The Python thin client supports all link:sql-reference[SQL commands] that are supported by Ignite.
The commands are executed via the `sql()` method of the cache object.
The `sql()` method returns a generator that yields the resulting rows.
-Refer to the link:sql-reference/sql-reference-overview[SQL Reference] section for the list of supported commands.
+Refer to the link:sql-reference[SQL Reference] section for the list of supported commands.
[source, python]
-------------------------------------------------------------------------------