http://git-wip-us.apache.org/repos/asf/drill/blob/baa4eddb/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md b/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md
index df2fcb9..9e5ad08 100644
--- a/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md
+++ b/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md
@@ -1,6 +1,6 @@
---
title: "Configuration Options Introduction"
-date: 2016-02-06 00:18:12 UTC
+date: 2016-08-04 22:49:29 UTC
parent: "Configuration Options"
---
Drill provides many configuration options that you can enable, disable, or
@@ -15,77 +15,79 @@ The sys.options table contains information about system and session options. The
## System Options
The sys.options table lists the following options that you can set as a system or session option as described in the section, ["Planning and Execution Options"]({{site.baseurl}}/docs/planning-and-execution-options).
-| Name | Default | Comments |
-|------------------------------------------------|------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| drill.exec.functions.cast_empty_string_to_null | FALSE | In a text file, treat empty fields as NULL values instead of empty string. |
-| drill.exec.storage.file.partition.column.label | dir | The column label for directory levels in results of queries of files in a directory. Accepts a string input. |
-| exec.enable_union_type | false | Enable support for Avro union type. |
-| exec.errors.verbose | FALSE | Toggles verbose output of executable error messages |
-| exec.java_compiler | DEFAULT | Switches between DEFAULT, JDK, and JANINO mode for the current session. Uses Janino by default for generated source code of less than exec.java_compiler_janino_maxsize; otherwise, switches to the JDK compiler. |
-| exec.java_compiler_debug | TRUE | Toggles the output of debug-level compiler error messages in runtime generated code. |
-| exec.java_compiler_janino_maxsize | 262144 | See the exec.java_compiler option comment. Accepts inputs of type LONG. |
-| exec.max_hash_table_size | 1073741824 | Ending size in buckets for hash tables. Range: 0 - 1073741824. |
-| exec.min_hash_table_size | 65536 | Starting size in bucketsfor hash tables. Increase according to available memory to improve performance. Increasing for very large aggregations or joins when you have large amounts of memory for Drill to use. Range: 0 - 1073741824. |
-| exec.queue.enable | FALSE | Changes the state of query queues. False allows unlimited concurrent queries. |
-| exec.queue.large | 10 | Sets the number of large queries that can run concurrently in the cluster. Range: 0-1000 |
-| exec.queue.small | 100 | Sets the number of small queries that can run concurrently in the cluster. Range: 0-1001 |
-| exec.queue.threshold | 30000000 | Sets the cost threshold, which depends on the complexity of the queries in queue, for determining whether query is large or small. Complex queries have higher thresholds. Range: 0-9223372036854775807 |
-| exec.queue.timeout_millis | 300000 | Indicates how long a query can wait in queue before the query fails. Range: 0-9223372036854775807 |
-| exec.schedule.assignment.old | FALSE | Used to prevent query failure when no work units are assigned to a minor fragment, particularly when the number of files is much larger than the number of leaf fragments. |
-| exec.storage.enable_new_text_reader | TRUE | Enables the text reader that complies with the RFC 4180 standard for text/csv files. |
-| new_view_default_permissions | 700 | Sets view permissions using an octal code in the Unix tradition. |
-| planner.add_producer_consumer | FALSE | Increase prefetching of data from disk. Disable for in-memory reads. |
-| planner.affinity_factor | 1.2 | Factor by which a node with endpoint affinity is favored while creating assignment. Accepts inputs of type DOUBLE. |
-| planner.broadcast_factor | 1 | A heuristic parameter for influencing the broadcast of records as part of a query. |
-| planner.broadcast_threshold | 10000000 | The maximum number of records allowed to be broadcast as part of a query. After one million records, Drill reshuffles data rather than doing a broadcast to one side of the join. Range: 0-2147483647 |
-| planner.disable_exchanges | FALSE | Toggles the state of hashing to a random exchange. |
-| planner.enable_broadcast_join | TRUE | Changes the state of aggregation and join operators. The broadcast join can be used for hash join, merge join and nested loop join. Use to join a large (fact) table to relatively smaller (dimension) tables. Do not disable. |
-| planner.enable_constant_folding | TRUE | If one side of a filter condition is a constant expression, constant folding evaluates the expression in the planning phase and replaces the expression with the constant value. For example, Drill can rewrite WHERE age + 5 < 42 as WHERE age < 37. |
-| planner.enable_decimal_data_type | FALSE | False disables the DECIMAL data type, including casting to DECIMAL and reading DECIMAL types from Parquet and Hive. |
-| planner.enable_demux_exchange | FALSE | Toggles the state of hashing to a demulitplexed exchange. |
-| planner.enable_hash_single_key | TRUE | Each hash key is associated with a single value. |
-| planner.enable_hashagg | TRUE | Enable hash aggregation; otherwise, Drill does a sort-based aggregation. Does not write to disk. Enable is recommended. |
-| planner.enable_hashjoin | TRUE | Enable the memory hungry hash join. Drill assumes that a query with have adequate memory to complete and tries to use the fastest operations possible to complete the planned inner, left, right, or full outer joins using a hash table. Does not write to disk. Disabling hash join allows Drill to manage arbitrarily large data in a small memory footprint. |
-| planner.enable_hashjoin_swap | TRUE | Enables consideration of multiple join order sequences during the planning phase. Might negatively affect the performance of some queries due to inaccuracy of estimated row count especially after a filter, join, or aggregation. |
-| planner.enable_hep_join_opt | | Enables the heuristic planner for joins. |
-| planner.enable_mergejoin | TRUE | Sort-based operation. A merge join is used for inner join, left and right outer joins. Inputs to the merge join must be sorted. It reads the sorted input streams from both sides and finds matching rows. Writes to disk. |
-| planner.enable_multiphase_agg | TRUE | Each minor fragment does a local aggregation in phase 1, distributes on a hash basis using GROUP-BY keys partially aggregated results to other fragments, and all the fragments perform a total aggregation using this data. |
-| planner.enable_mux_exchange | TRUE | Toggles the state of hashing to a multiplexed exchange. |
-| planner.enable_nestedloopjoin | TRUE | Sort-based operation. Writes to disk. |
-| planner.enable_nljoin_for_scalar_only | TRUE | Supports nested loop join planning where the right input is scalar in order to enable NOT-IN, Inequality, Cartesian, and uncorrelated EXISTS planning. |
-| planner.enable_streamagg | TRUE | Sort-based operation. Writes to disk. |
-| planner.identifier_max_length | 1024 | A minimum length is needed because option names are identifiers themselves. |
-| planner.join.hash_join_swap_margin_factor | 10 | The number of join order sequences to consider during the planning phase. |
-| planner.join.row_count_estimate_factor | 1 | The factor for adjusting the estimated row count when considering multiple join order sequences during the planning phase. |
-| planner.memory.average_field_width | 8 | Used in estimating memory requirements. |
-| planner.memory.enable_memory_estimation | FALSE | Toggles the state of memory estimation and re-planning of the query. When enabled, Drill conservatively estimates memory requirements and typically excludes these operators from the plan and negatively impacts performance. |
-| planner.memory.hash_agg_table_factor | 1.1 | A heuristic value for influencing the size of the hash aggregation table. |
-| planner.memory.hash_join_table_factor | 1.1 | A heuristic value for influencing the size of the hash aggregation table. |
-| planner.memory.max_query_memory_per_node | 2147483648 bytes | Sets the maximum amount of direct memory allocated to the sort operator in each query on a node. If a query plan contains multiple sort operators, they all share this memory. If you encounter memory issues when running queries with sort operators, increase the value of this option. |
-| planner.memory.non_blocking_operators_memory | 64 | Extra query memory per node for non-blocking operators. This option is currently used only for memory estimation. Range: 0-2048 MB |
-| planner.memory_limit | 268435456 bytes | Defines the maximum amount of direct memory allocated to a query for planning. When multiple queries run concurrently, each query is allocated the amount of memory set by this parameter.Increase the value of this parameter and rerun the query if partition pruning failed due to insufficient memory. |
-| planner.nestedloopjoin_factor | 100 | A heuristic value for influencing the nested loop join. |
-| planner.partitioner_sender_max_threads | 8 | Upper limit of threads for outbound queuing. |
-| planner.partitioner_sender_set_threads | -1 | Overwrites the number of threads used to send out batches of records. Set to -1 to disable. Typically not changed. |
-| planner.partitioner_sender_threads_factor | 2 | A heuristic param to use to influence final number of threads. The higher the value the fewer the number of threads. |
-| planner.producer_consumer_queue_size | 10 | How much data to prefetch from disk in record batches out-of-band of query execution. The larger the queue size, the greater the amount of memory that the queue and overall query execution consumes. |
-| planner.slice_target | 100000 | The number of records manipulated within a fragment before Drill parallelizes operations. |
-| planner.width.max_per_node | 3 | Maximum number of threads that can run in parallel for a query on a node. A slice is an individual thread. This number indicates the maximum number of slices per query for the query\u2019s major fragment on a node. |
-| planner.width.max_per_query | 1000 | Same as max per node but applies to the query as executed by the entire cluster. For example, this value might be the number of active Drillbits, or a higher number to return results faster. |
-| security.admin.user_groups | n/a | Unsupported as of 1.4. A comma-separated list of administrator groups for Web Console security. |
-| security.admin.users | <a name> | Unsupported as of 1.4. A comma-separated list of user names who you want to give administrator privileges. |
-| store.format | parquet | Output format for data written to tables with the CREATE TABLE AS (CTAS) command. Allowed values are parquet, json, psv, csv, or tsv. |
-| store.hive.optimize_scan_with_native_readers | FALSE | Optimize reads of Parquet-backed external tables from Hive by using Drill native readers instead of the Hive Serde interface. (Drill 1.2 and later) |
-| store.json.all_text_mode | FALSE | Drill reads all data from the JSON files as VARCHAR. Prevents schema change errors. |
-| store.json.extended_types | FALSE | Turns on special JSON structures that Drill serializes for storing more type information than the four basic JSON types. |
-| store.json.read_numbers_as_double | FALSE | Reads numbers with or without a decimal point as DOUBLE. Prevents schema change errors. |
-| store.mongo.all_text_mode | FALSE | Similar to store.json.all_text_mode for MongoDB. |
-| store.mongo.read_numbers_as_double | FALSE | Similar to store.json.read_numbers_as_double. |
-| store.parquet.block-size | 536870912 | Sets the size of a Parquet row group to the number of bytes less than or equal to the block size of MFS, HDFS, or the file system. |
-| store.parquet.compression | snappy | Compression type for storing Parquet output. Allowed values: snappy, gzip, none |
-| store.parquet.enable_dictionary_encoding | FALSE | For internal use. Do not change. |
-| store.parquet.dictionary.page-size | 1048576 | |
-| store.parquet.use_new_reader | FALSE | Not supported in this release. |
-| store.partition.hash_distribute | FALSE | Uses a hash algorithm to distribute data on partition keys in a CTAS partitioning operation. An alpha option--for experimental use at this stage. Do not use in production systems. |
-| store.text.estimated_row_size_bytes | 100 | Estimate of the row size in a delimited text file, such as csv. The closer to actual, the better the query plan. Used for all csv files in the system/session where the value is set. Impacts the decision to plan a broadcast join or not. |
-| window.enable | TRUE | Enable or disable window functions in Drill 1.1 and later. |
+| Name | Default | Comments
|
+|------------------------------------------------|--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| drill.exec.functions.cast_empty_string_to_null | FALSE | In a text file, treat empty fields as NULL values instead of empty string.
|
+| drill.exec.storage.file.partition.column.label | dir | The column label for directory levels in results of queries of files in a directory. Accepts a string input.
|
+| exec.enable_union_type | FALSE | Enable support for Avro union type.
|
+| exec.errors.verbose | FALSE | Toggles verbose output of executable error messages
|
+| exec.java_compiler | DEFAULT | Switches between DEFAULT, JDK, and JANINO mode for the current session. Uses Janino by default for generated source code of less than exec.java_compiler_janino_maxsize; otherwise, switches to the JDK compiler.
|
+| exec.java_compiler_debug | TRUE | Toggles the output of debug-level compiler error messages in runtime generated code.
|
+| exec.java_compiler_janino_maxsize | 262144 | See the exec.java_compiler option comment. Accepts inputs of type LONG.
|
+| exec.max_hash_table_size | 1073741824 | Ending size in buckets for hash tables. Range: 0 - 1073741824.
|
+| exec.min_hash_table_size | 65536 | Starting size in bucketsfor hash tables. Increase according to available memory to improve performance. Increasing for very large aggregations or joins when you have large amounts of memory for Drill to use. Range: 0 - 1073741824.
|
+| exec.queue.enable | FALSE | Changes the state of query queues. False allows unlimited concurrent queries.
|
+| exec.queue.large | 10 | Sets the number of large queries that can run concurrently in the cluster. Range: 0-1000
|
+| exec.queue.small | 100 | Sets the number of small queries that can run concurrently in the cluster. Range: 0-1001
|
+| exec.queue.threshold | 30000000 | Sets the cost threshold, which depends on the complexity of the queries in queue, for determining whether query is large or small. Complex queries have higher thresholds. Range: 0-9223372036854775807
|
+| exec.queue.timeout_millis | 300000 | Indicates how long a query can wait in queue before the query fails. Range: 0-9223372036854775807
|
+| exec.schedule.assignment.old | FALSE | Used to prevent query failure when no work units are assigned to a minor fragment, particularly when the number of files is much larger than the number of leaf fragments.
|
+| exec.storage.enable_new_text_reader | TRUE | Enables the text reader that complies with the RFC 4180 standard for text/csv files.
|
+| new_view_default_permissions | 700 | Sets view permissions using an octal code in the Unix tradition.
|
+| planner.add_producer_consumer | FALSE | Increase prefetching of data from disk. Disable for in-memory reads.
|
+| planner.affinity_factor | 1.2 | Factor by which a node with endpoint affinity is favored while creating assignment. Accepts inputs of type DOUBLE.
|
+| planner.broadcast_factor | 1 | A heuristic parameter for influencing the broadcast of records as part of a query.
|
+| planner.broadcast_threshold | 10000000 | The maximum number of records allowed to be broadcast as part of a query. After one million records, Drill reshuffles data rather than doing a broadcast to one side of the join. Range: 0-2147483647
|
+| planner.disable_exchanges | FALSE | Toggles the state of hashing to a random exchange.
|
+| planner.enable_broadcast_join | TRUE | Changes the state of aggregation and join operators. The broadcast join can be used for hash join, merge join and nested loop join. Use to join a large (fact) table to relatively smaller (dimension) tables. Do not disable.
|
+| planner.enable_constant_folding | TRUE | If one side of a filter condition is a constant expression, constant folding evaluates the expression in the planning phase and replaces the expression with the constant value. For example, Drill can rewrite WHERE age + 5 < 42 as WHERE age < 37.
|
+| planner.enable_decimal_data_type | FALSE | False disables the DECIMAL data type, including casting to DECIMAL and reading DECIMAL types from Parquet and Hive.
|
+| planner.enable_demux_exchange | FALSE | Toggles the state of hashing to a demulitplexed exchange.
|
+| planner.enable_hash_single_key | TRUE | Each hash key is associated with a single value.
|
+| planner.enable_hashagg | TRUE | Enable hash aggregation; otherwise, Drill does a sort-based aggregation. Does not write to disk. Enable is recommended.
|
+| planner.enable_hashjoin | TRUE | Enable the memory hungry hash join. Drill assumes that a query with have adequate memory to complete and tries to use the fastest operations possible to complete the planned inner, left, right, or full outer joins using a hash table. Does not write to disk. Disabling hash join allows Drill to manage arbitrarily large data in a small memory footprint.
|
+| planner.enable_hashjoin_swap | TRUE | Enables consideration of multiple join order sequences during the planning phase. Might negatively affect the performance of some queries due to inaccuracy of estimated row count especially after a filter, join, or aggregation.
|
+| planner.enable_hep_join_opt | | Enables the heuristic planner for joins.
|
+| planner.enable_mergejoin | TRUE | Sort-based operation. A merge join is used for inner join, left and right outer joins. Inputs to the merge join must be sorted. It reads the sorted input streams from both sides and finds matching rows. Writes to disk.
|
+| planner.enable_multiphase_agg | TRUE | Each minor fragment does a local aggregation in phase 1, distributes on a hash basis using GROUP-BY keys partially aggregated results to other fragments, and all the fragments perform a total aggregation using this data.
|
+| planner.enable_mux_exchange | TRUE | Toggles the state of hashing to a multiplexed exchange.
|
+| planner.enable_nestedloopjoin | TRUE | Sort-based operation. Writes to disk.
|
+| planner.enable_nljoin_for_scalar_only | TRUE | Supports nested loop join planning where the right input is scalar in order to enable NOT-IN, Inequality, Cartesian, and uncorrelated EXISTS planning.
|
+| planner.enable_streamagg | TRUE | Sort-based operation. Writes to disk.
|
+| planner.filter.max_selectivity_estimate_factor | - | Sets the maximum filter selectivity estimate. The selectivity can vary between 0 and 1. For more details, see planner.filter.min_selectivity_estimate_factor.
|
+| planner.filter.min_selectivity_estimate_factor | - | Sets the minimum filter selectivity estimate to increase the parallelization of the major fragment performing a join. This option is useful for deeply nested queries with complicated predicates and serves as a workaround when statistics are insufficient or unavailable. The selectivity can vary between 0 and 1. The value of this option caps the estimated SELECTIVITY. The estimated ROWCOUNT is derived by multiplying the estimated SELECTIVITY by the estimated ROWCOUNT of the upstream operator. The estimated ROWCOUNT displays when you use the EXPLAIN PLAN INCLUDING ALL ATTRIBUTES FOR command. This option does not control the estimated ROWCOUNT of downstream operators (post FILTER). However, estimated ROWCOUNTs may change because the operator ROWCOUNTs depend on their downstream operators. The FILTER operator relies on the input of its immediate upstream operator, for example SCAN, AGGREGATE.
If two filters are present in a plan, each filter may have a different estimated ROWCOUNT based on the immediate upstream operator's estimated ROWCOUNT. |
+| planner.identifier_max_length | 1024 | A minimum length is needed because option names are identifiers themselves.
|
+| planner.join.hash_join_swap_margin_factor | 10 | The number of join order sequences to consider during the planning phase.
|
+| planner.join.row_count_estimate_factor | 1 | The factor for adjusting the estimated row count when considering multiple join order sequences during the planning phase.
|
+| planner.memory.average_field_width | 8 | Used in estimating memory requirements.
|
+| planner.memory.enable_memory_estimation | FALSE | Toggles the state of memory estimation and re-planning of the query. When enabled, Drill conservatively estimates memory requirements and typically excludes these operators from the plan and negatively impacts performance.
|
+| planner.memory.hash_agg_table_factor | 1.1 | A heuristic value for influencing the size of the hash aggregation table.
|
+| planner.memory.hash_join_table_factor | 1.1 | A heuristic value for influencing the size of the hash aggregation table.
|
+| planner.memory.max_query_memory_per_node | 2147483648 bytes | Sets the maximum amount of direct memory allocated to the sort operator in each query on a node. If a query plan contains multiple sort operators, they all share this memory. If you encounter memory issues when running queries with sort operators, increase the value of this option.
<TRUNCATED>