You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2020/04/07 12:57:48 UTC
[spark] branch branch-3.0 updated (afab532 -> 304f7f2)
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git.
from afab532 [SPARK-31002][CORE][DOC][FOLLOWUP] Add version information to the configuration of Core
new dec53be [SPARK-31295][DOC] Supplement version for configuration appear in doc
new 8af58eb [SPARK-31295][DOC][FOLLOWUP] Supplement version for configuration appear in doc
new 70bf2ff [SPARK-31282][DOC] Supplement version for configuration appear in security doc
new 304f7f2 [SPARK-31269][DOC] Supplement version for configuration only appear in configuration doc
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/configuration.md | 88 +++++++++++++++++++++++++++++++---------
docs/security.md | 43 ++++++++++++++------
docs/spark-standalone.md | 13 +++++-
docs/sql-data-sources-avro.md | 21 ++++++++--
docs/sql-data-sources-orc.md | 16 ++++++--
docs/sql-data-sources-parquet.md | 9 +++-
docs/sql-performance-tuning.md | 30 ++++++++++----
docs/sql-ref-ansi-compliance.md | 4 +-
8 files changed, 175 insertions(+), 49 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 03/04: [SPARK-31282][DOC] Supplement version for
configuration appear in security doc
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 70bf2ff09d3d71d8fb9dcee7f2298288caa641e7
Author: beliefer <be...@163.com>
AuthorDate: Tue Mar 31 12:33:01 2020 +0900
[SPARK-31282][DOC] Supplement version for configuration appear in security doc
### What changes were proposed in this pull request?
This PR supplements version for configuration appear in security doc.
I sorted out some information show below.
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.network.crypto.keyLength | 2.2.0 | SPARK-19139 | 8f3f73abc1fe62496722476460c174af0250e3fe#diff-0ac65da2bc6b083fb861fe410c7688c2 |
spark.network.crypto.keyFactoryAlgorithm | 2.2.0 | SPARK-19139 | 8f3f73abc1fe62496722476460c174af0250e3fe#diff-0ac65da2bc6b083fb861fe410c7688c2 |
spark.network.crypto.config.* | 2.2.0 | SPARK-19139 | 8f3f73abc1fe62496722476460c174af0250e3fe#diff-0ac65da2bc6b083fb861fe410c7688c2 |
spark.network.crypto.saslFallback | 2.2.0 | SPARK-19139 | 8f3f73abc1fe62496722476460c174af0250e3fe#diff-0ac65da2bc6b083fb861fe410c7688c2 |
spark.authenticate.enableSaslEncryption | 2.2.0 | SPARK-19139 | 8f3f73abc1fe62496722476460c174af0250e3fe#diff-0ac65da2bc6b083fb861fe410c7688c2 |
spark.network.sasl.serverAlwaysEncrypt | 1.4.0 | SPARK-6229 | 38d4e9e446b425ca6a8fe8d8080f387b08683842#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 |
spark.ui.filters | 1.0.0 | SPARK-1189 | 7edbea41b43e0dc11a2de156be220db8b7952d01#diff-f79a5ead735b3d0b34b6b94486918e1c |
spark.acls.enable | 1.1.0 | SPARK-1890 and SPARK-1891 | e3fe6571decfdc406ec6d505fd92f9f2b85a618c#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.ui.view.acls | 1.0.0 | SPARK-1189 | 7edbea41b43e0dc11a2de156be220db8b7952d01#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.ui.view.acls.groups | 2.0.0 | SPARK-4224 | ae79032dcf160796851ca29116cca146c4d86ada#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.admin.acls | 1.1.0 | SPARK-1890 and SPARK-1891 | e3fe6571decfdc406ec6d505fd92f9f2b85a618c#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.admin.acls.groups | 2.0.0 | SPARK-4224 | ae79032dcf160796851ca29116cca146c4d86ada#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.modify.acls | 1.1.0 | SPARK-1890 and SPARK-1891 | e3fe6571decfdc406ec6d505fd92f9f2b85a618c#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.modify.acls.groups | 2.0.0 | SPARK-4224 | ae79032dcf160796851ca29116cca146c4d86ada#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.user.groups.mapping | 2.0.0 | SPARK-4224 | ae79032dcf160796851ca29116cca146c4d86ada#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.history.ui.acls.enable | 1.0.1 | Spark 1489 | c8dd13221215275948b1a6913192d40e0c8cbadd#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.ui.admin.acls | 2.1.1 | SPARK-19033 | 4ca1788805e4a0131ba8f0ccb7499ee0e0242837#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.ui.admin.acls.groups | 2.1.1 | SPARK-19033 | 4ca1788805e4a0131ba8f0ccb7499ee0e0242837#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.ui.xXssProtection | 2.3.0 | SPARK-22188 | 5a07aca4d464e96d75ea17bf6768e24b829872ec#diff-6bdad48cfc34314e89599655442ff210 |
spark.ui.xContentTypeOptions.enabled | 2.3.0 | SPARK-22188 | 5a07aca4d464e96d75ea17bf6768e24b829872ec#diff-6bdad48cfc34314e89599655442ff210 |
spark.ui.strictTransportSecurity | 2.3.0 | SPARK-22188 | 5a07aca4d464e96d75ea17bf6768e24b829872ec#diff-6bdad48cfc34314e89599655442ff210 |
spark.security.credentials.${service}.enabled | 2.3.0 | SPARK-20434 | a18d637112b97d2caaca0a8324bdd99086664b24#diff-da6c1fd6d8b0c7538a3e77a09e06a083 |
spark.kerberos.access.hadoopFileSystems | 3.0.0 | SPARK-26766 | d0443a74d185ec72b747fa39994fa9a40ce974cf#diff-6bdad48cfc34314e89599655442ff210 |
### Why are the changes needed?
Supplemental configuration version information.
### Does this PR introduce any user-facing change?
'No'.
### How was this patch tested?
Jenkins test
Closes #28044 from beliefer/supplement-version-to-security-doc.
Authored-by: beliefer <be...@163.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
docs/security.md | 43 +++++++++++++++++++++++++++++++------------
1 file changed, 31 insertions(+), 12 deletions(-)
diff --git a/docs/security.md b/docs/security.md
index 5496879..aef6e69 100644
--- a/docs/security.md
+++ b/docs/security.md
@@ -158,7 +158,7 @@ The following table describes the different options available for configuring th
<td>
The length in bits of the encryption key to generate. Valid values are 128, 192 and 256.
</td>
- <td></td>
+ <td>2.2.0</td>
</tr>
<tr>
<td><code>spark.network.crypto.keyFactoryAlgorithm</code></td>
@@ -167,7 +167,7 @@ The following table describes the different options available for configuring th
The key factory algorithm to use when generating encryption keys. Should be one of the
algorithms supported by the javax.crypto.SecretKeyFactory class in the JRE being used.
</td>
- <td></td>
+ <td>2.2.0</td>
</tr>
<tr>
<td><code>spark.network.crypto.config.*</code></td>
@@ -177,7 +177,7 @@ The following table describes the different options available for configuring th
use. The config name should be the name of commons-crypto configuration without the
<code>commons.crypto</code> prefix.
</td>
- <td></td>
+ <td>2.2.0</td>
</tr>
<tr>
<td><code>spark.network.crypto.saslFallback</code></td>
@@ -196,6 +196,7 @@ The following table describes the different options available for configuring th
<td>
Enable SASL-based encrypted communication.
</td>
+ <td>2.2.0</td>
</tr>
<tr>
<td><code>spark.network.sasl.serverAlwaysEncrypt</code></td>
@@ -204,6 +205,7 @@ The following table describes the different options available for configuring th
Disable unencrypted connections for ports using SASL authentication. This will deny connections
from clients that have authentication enabled, but do not request SASL-based encryption.
</td>
+ <td>1.4.0</td>
</tr>
</table>
@@ -286,7 +288,7 @@ below.
The following options control the authentication of Web UIs:
<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.ui.filters</code></td>
<td>None</td>
@@ -294,6 +296,7 @@ The following options control the authentication of Web UIs:
See the <a href="configuration.html#spark-ui">Spark UI</a> configuration for how to configure
filters.
</td>
+ <td>1.0.0</td>
</tr>
<tr>
<td><code>spark.acls.enable</code></td>
@@ -303,6 +306,7 @@ The following options control the authentication of Web UIs:
permissions to view or modify the application. Note this requires the user to be authenticated,
so if no authentication filter is installed, this option does not do anything.
</td>
+ <td>1.1.0</td>
</tr>
<tr>
<td><code>spark.admin.acls</code></td>
@@ -310,6 +314,7 @@ The following options control the authentication of Web UIs:
<td>
Comma-separated list of users that have view and modify access to the Spark application.
</td>
+ <td>1.1.0</td>
</tr>
<tr>
<td><code>spark.admin.acls.groups</code></td>
@@ -317,6 +322,7 @@ The following options control the authentication of Web UIs:
<td>
Comma-separated list of groups that have view and modify access to the Spark application.
</td>
+ <td>2.0.0</td>
</tr>
<tr>
<td><code>spark.modify.acls</code></td>
@@ -324,6 +330,7 @@ The following options control the authentication of Web UIs:
<td>
Comma-separated list of users that have modify access to the Spark application.
</td>
+ <td>1.1.0</td>
</tr>
<tr>
<td><code>spark.modify.acls.groups</code></td>
@@ -331,6 +338,7 @@ The following options control the authentication of Web UIs:
<td>
Comma-separated list of groups that have modify access to the Spark application.
</td>
+ <td>2.0.0</td>
</tr>
<tr>
<td><code>spark.ui.view.acls</code></td>
@@ -338,6 +346,7 @@ The following options control the authentication of Web UIs:
<td>
Comma-separated list of users that have view access to the Spark application.
</td>
+ <td>1.0.0</td>
</tr>
<tr>
<td><code>spark.ui.view.acls.groups</code></td>
@@ -345,6 +354,7 @@ The following options control the authentication of Web UIs:
<td>
Comma-separated list of groups that have view access to the Spark application.
</td>
+ <td>2.0.0</td>
</tr>
<tr>
<td><code>spark.user.groups.mapping</code></td>
@@ -361,6 +371,7 @@ The following options control the authentication of Web UIs:
Windows environment is currently <b>not</b> supported. However, a new platform/protocol can
be supported by implementing the trait mentioned above.
</td>
+ <td>2.0.0</td>
</tr>
</table>
@@ -375,7 +386,7 @@ servlet filters.
To enable authorization in the SHS, a few extra options are used:
<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.history.ui.acls.enable</code></td>
<td>false</td>
@@ -389,6 +400,7 @@ To enable authorization in the SHS, a few extra options are used:
If disabled, no access control checks are made for any application UIs available through
the history server.
</td>
+ <td>1.0.1</td>
</tr>
<tr>
<td><code>spark.history.ui.admin.acls</code></td>
@@ -397,6 +409,7 @@ To enable authorization in the SHS, a few extra options are used:
Comma separated list of users that have view access to all the Spark applications in history
server.
</td>
+ <td>2.1.1</td>
</tr>
<tr>
<td><code>spark.history.ui.admin.acls.groups</code></td>
@@ -405,6 +418,7 @@ To enable authorization in the SHS, a few extra options are used:
Comma separated list of groups that have view access to all the Spark applications in history
server.
</td>
+ <td>2.1.1</td>
</tr>
</table>
@@ -620,7 +634,7 @@ Apache Spark can be configured to include HTTP headers to aid in preventing Cros
Security.
<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.ui.xXssProtection</code></td>
<td><code>1; mode=block</code></td>
@@ -635,6 +649,7 @@ Security.
of the page if an attack is detected.)</li>
</ul>
</td>
+ <td>2.3.0</td>
</tr>
<tr>
<td><code>spark.ui.xContentTypeOptions.enabled</code></td>
@@ -642,7 +657,8 @@ Security.
<td>
When enabled, X-Content-Type-Options HTTP response header will be set to "nosniff".
</td>
- </tr>
+ <td>2.3.0</td>
+</tr>
<tr>
<td><code>spark.ui.strictTransportSecurity</code></td>
<td>None</td>
@@ -656,6 +672,7 @@ Security.
<li><code>max-age=<expire-time>; preload</code></li>
</ul>
</td>
+ <td>2.3.0</td>
</tr>
</table>
@@ -796,16 +813,17 @@ deployment-specific page for more information.
The following options provides finer-grained control for this feature:
<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.security.credentials.${service}.enabled</code></td>
<td><code>true</code></td>
<td>
- Controls whether to obtain credentials for services when security is enabled.
- By default, credentials for all supported services are retrieved when those services are
- configured, but it's possible to disable that behavior if it somehow conflicts with the
- application being run.
+ Controls whether to obtain credentials for services when security is enabled.
+ By default, credentials for all supported services are retrieved when those services are
+ configured, but it's possible to disable that behavior if it somehow conflicts with the
+ application being run.
</td>
+ <td>2.3.0</td>
</tr>
<tr>
<td><code>spark.kerberos.access.hadoopFileSystems</code></td>
@@ -818,6 +836,7 @@ The following options provides finer-grained control for this feature:
or in a trusted realm). Spark acquires security tokens for each of the filesystems so that
the Spark application can access those remote Hadoop filesystems.
</td>
+ <td>3.0.0</td>
</tr>
</table>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 04/04: [SPARK-31269][DOC] Supplement version for
configuration only appear in configuration doc
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 304f7f293068bdb3b55208e473ee01c256c8bd2a
Author: beliefer <be...@163.com>
AuthorDate: Tue Mar 31 12:32:04 2020 +0900
[SPARK-31269][DOC] Supplement version for configuration only appear in configuration doc
### What changes were proposed in this pull request?
The `configuration.md` exists some config not organized by `ConfigEntry`.
This PR supplements version for configuration only appear in configuration doc.
I sorted out some information show below.
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.app.name | 0.9.0 | None | 994f080f8ae3372366e6004600ba791c8a372ff0#diff-529fc5c06b9731c1fbda6f3db60b16aa |
spark.driver.resource.{resourceName}.amount | 3.0.0 | SPARK-27760 | d30284b5a51dd784f663eb4eea37087b35a54d00#diff-76e731333fb756df3bff5ddb3b731c46 |
spark.driver.resource.{resourceName}.discoveryScript | 3.0.0 | SPARK-27488 | 74e5e41eebf9ed596b48e6db52a2a9c642e5cbc3#diff-76e731333fb756df3bff5ddb3b731c46 |
spark.driver.resource.{resourceName}.vendor | 3.0.0 | SPARK-27362 | 1277f8fa92da85d9e39d9146e3099fcb75c71a8f#diff-76e731333fb756df3bff5ddb3b731c46 |
spark.executor.resource.{resourceName}.amount | 3.0.0 | SPARK-27760 | d30284b5a51dd784f663eb4eea37087b35a54d00#diff-76e731333fb756df3bff5ddb3b731c46 |
spark.executor.resource.{resourceType}.discoveryScript | 3.0.0 | SPARK-27024 | db2e3c43412e4a7fb4a46c58d73d9ab304a1e949#diff-76e731333fb756df3bff5ddb3b731c46 |
spark.executor.resource.{resourceName}.vendor | 3.0.0 | SPARK-27362 | 1277f8fa92da85d9e39d9146e3099fcb75c71a8f#diff-76e731333fb756df3bff5ddb3b731c46 |
spark.local.dir | 0.5.0 | None | 0e93891d3d7df849cff6442038c111ffd42a5243#diff-17fd275d280b667722664ed833c6402a |
spark.logConf | 0.9.0 | None | d8bcc8e9a095c1b20dd7a17b6535800d39bff80e#diff-364713d7776956cb8b0a771e9b62f82d |
spark.master | 0.9.0 | SPARK-544 | 2573add94cf920a88f74d80d8ea94218d812704d#diff-529fc5c06b9731c1fbda6f3db60b16aa |
spark.driver.defaultJavaOptions | 3.0.0 | SPARK-23472 | f83000597f250868de9722d8285fed013abc5ecf#diff-a78ecfc6a89edfaf0b60a5eaa0381970 |
spark.executor.defaultJavaOptions | 3.0.0 | SPARK-23472 | f83000597f250868de9722d8285fed013abc5ecf#diff-a78ecfc6a89edfaf0b60a5eaa0381970 |
spark.executorEnv.[EnvironmentVariableName] | 0.9.0 | None | 642029e7f43322f84abe4f7f36bb0b1b95d8101d#diff-529fc5c06b9731c1fbda6f3db60b16aa |
spark.python.profile | 1.2.0 | SPARK-3478 | 1aa549ba9839565274a12c52fa1075b424f138a6#diff-d6fe2792e44f6babc94aabfefc8b9bce |
spark.python.profile.dump | 1.2.0 | SPARK-3478 | 1aa549ba9839565274a12c52fa1075b424f138a6#diff-d6fe2792e44f6babc94aabfefc8b9bce |
spark.python.worker.memory | 1.1.0 | SPARK-2538 | 14174abd421318e71c16edd24224fd5094bdfed4#diff-d6fe2792e44f6babc94aabfefc8b9bce |
spark.jars.packages | 1.5.0 | SPARK-9263 | 34335719a372c1951fdb4dd25b75b086faf1076f#diff-63a5d817d2d45ae24de577f6a1bd80f9 |
spark.jars.excludes | 1.5.0 | SPARK-9263 | 34335719a372c1951fdb4dd25b75b086faf1076f#diff-63a5d817d2d45ae24de577f6a1bd80f9 |
spark.jars.ivy | 1.3.0 | SPARK-5341 | 3b7acd22ab4a134c74746e3b9a803dbd34d43855#diff-63a5d817d2d45ae24de577f6a1bd80f9 |
spark.jars.ivySettings | 2.2.0 | SPARK-17568 | 3bc2eff8880a3ba8d4318118715ea1a47048e3de#diff-4d2ab44195558d5a9d5f15b8803ef39d |
spark.jars.repositories | 2.3.0 | SPARK-21403 | d8257b99ddae23f702f312640a5335ddb4554403#diff-4d2ab44195558d5a9d5f15b8803ef39d |
spark.shuffle.io.maxRetries | 1.2.0 | SPARK-4188 | c1ea5c542f3267c0b23a7775887e3a6ece793fe3#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 |
spark.shuffle.io.numConnectionsPerPeer | 1.2.1 | SPARK-4740 | 441ec3451730c7ae3dbef8952e313071d6147ab6#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 |
spark.shuffle.io.preferDirectBufs | 1.2.0 | SPARK-4188 | c1ea5c542f3267c0b23a7775887e3a6ece793fe3#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 |
spark.shuffle.io.retryWait | 1.2.1 | None | 5e5d8f469a1bea9bbe606f772ccdcab7c184c651#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 |
spark.shuffle.io.backLog | 1.1.1 | SPARK-2468 | 66b4c81db7e826c00f7fb449b8a8af810cf7dd9a#diff-bdee8e601924d41e93baa7287189e878 |
spark.shuffle.service.index.cache.size | 2.3.0 | SPARK-21501 | 1662e93119d68498942386906de309d35f4a135f#diff-97d5edc927a83a678e013ae00343df94 |
spark.shuffle.maxChunksBeingTransferred | 2.3.0 | SPARK-21175 | 799e13161e89f1ea96cb1bc7b507a05af2e89cd0#diff-0ac65da2bc6b083fb861fe410c7688c2 |
spark.sql.ui.retainedExecutions | 1.5.0 | SPARK-8861 and SPARK-8862 | ebc3aad272b91cf58e2e1b4aa92b49b8a947a045#diff-81764e4d52817f83bdd5336ef1226bd9 |
spark.streaming.ui.retainedBatches | 1.0.0 | SPARK-1386 | f36dc3fed0a0671b0712d664db859da28c0a98e2#diff-56b8d67d07284cfab165d5363bd3500e |
spark.default.parallelism | 0.5.0 | None | e5c4cd8a5e188592f8786a265c0cd073c69ac886#diff-0544ebf7533fa70ff5103e0fe1f0b036 |
spark.files.fetchTimeout | 1.0.0 | None | f6f9d02e85d17da2f742ed0062f1648a9293e73c#diff-d239aee594001f8391676e1047a0381e |
spark.files.useFetchCache | 1.2.2 | SPARK-6313 | a2a94a154bdd00753b8d5e344d712664c7151050#diff-d239aee594001f8391676e1047a0381e |
spark.files.overwrite | 1.0.0 | None | 84670f2715392859624df290c1b52eb4ed4a9cb1#diff-d239aee594001f8391676e1047a0381e | Exists in branch-1.0, but the version of pom is 0.9.0-incubating-SNAPSHOT
spark.hadoop.cloneConf | 1.0.3 | SPARK-2546 | 6d8f1dd15afdc7432b5721c89f9b2b402460322b#diff-83eb37f7b0ebed3c14ccb7bff0d577c2 |
spark.hadoop.validateOutputSpecs | 1.0.1 | SPARK-1677 | 8100cbdb7546e8438019443cfc00683017c81278#diff-f70e97c099b5eac05c75288cb215e080 |
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version | 2.2.0 | SPARK-20107 | edc87d76efea7b4d19d9d0c4ddba274a3ccb8752#diff-76e731333fb756df3bff5ddb3b731c46 |
spark.rpc.io.backLog | 3.0.0 | SPARK-27868 | 09ed64d795d3199a94e175273fff6fcea6b52131#diff-76e731333fb756df3bff5ddb3b731c46 |
spark.network.io.preferDirectBufs | 3.0.0 | SPARK-24920 | e103c4a5e72bab8862ff49d6d4c1e62e642fc412#diff-0ac65da2bc6b083fb861fe410c7688c2 |
spark.port.maxRetries | 1.1.1 | SPARK-3565 | 32f2222e915f31422089139944a077e2cbd442f9#diff-d239aee594001f8391676e1047a0381e |
spark.core.connection.ack.wait.timeout | 1.1.1 | SPARK-2677 | bd3ce2ffb8964abb4d59918ebb2c230fe4614aa2#diff-f748e95f2aa97ed715afa53ddeeac9de |
spark.scheduler.listenerbus.eventqueue.shared.capacity | 3.0.0 | SPARK-28574 | c212c9d9ed7375cd1ea16c118733edd84037ec0d#diff-eb519ad78cc3cf0b95839cc37413b509 |
spark.scheduler.listenerbus.eventqueue.appStatus.capacity | 3.0.0 | SPARK-28574 | c212c9d9ed7375cd1ea16c118733edd84037ec0d#diff-eb519ad78cc3cf0b95839cc37413b509 |
spark.scheduler.listenerbus.eventqueue.executorManagement.capacity | 3.0.0 | SPARK-28574 | c212c9d9ed7375cd1ea16c118733edd84037ec0d#diff-eb519ad78cc3cf0b95839cc37413b509 |
spark.scheduler.listenerbus.eventqueue.eventLog.capacity | 3.0.0 | SPARK-28574 | c212c9d9ed7375cd1ea16c118733edd84037ec0d#diff-eb519ad78cc3cf0b95839cc37413b509 |
spark.scheduler.listenerbus.eventqueue.streams.capacity | 3.0.0 | SPARK-28574 | c212c9d9ed7375cd1ea16c118733edd84037ec0d#diff-eb519ad78cc3cf0b95839cc37413b509 |
spark.task.resource.{resourceName}.amount | 3.0.0 | SPARK-27760 | d30284b5a51dd784f663eb4eea37087b35a54d00#diff-76e731333fb756df3bff5ddb3b731c46 |
spark.stage.maxConsecutiveAttempts | 2.2.0 | SPARK-13369 | 7b5d873aef672aa0aee41e338bab7428101e1ad3#diff-6a9ff7fb74fd490a50462d45db2d5e11 |
spark.{driver\|executor}.rpc.io.serverThreads | 1.6.0 | SPARK-10745 | 7c5b641808740ba5eed05ba8204cdbaf3fc579f5#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 |
spark.{driver\|executor}.rpc.io.clientThreads | 1.6.0 | SPARK-10745 | 7c5b641808740ba5eed05ba8204cdbaf3fc579f5#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 |
spark.{driver\|executor}.rpc.netty.dispatcher.numThreads | 3.0.0 | SPARK-29398 | 2f0a38cb50e3e8b4b72219c7b2b8b15d51f6b931#diff-a68a21481fea5053848ca666dd3201d8 |
spark.r.driver.command | 1.5.3 | SPARK-10971 | 9695f452e86a88bef3bcbd1f3c0b00ad9e9ac6e1#diff-025470e1b7094d7cf4a78ea353fb3981 |
spark.r.shell.command | 2.1.0 | SPARK-17178 | fa6347938fc1c72ddc03a5f3cd2e929b5694f0a6#diff-a78ecfc6a89edfaf0b60a5eaa0381970 |
spark.graphx.pregel.checkpointInterval | 2.2.0 | SPARK-5484 | f971ce5dd0788fe7f5d2ca820b9ea3db72033ddc#diff-e399679417ffa6eeedf26a7630baca16 |
### Why are the changes needed?
Supplemental configuration version information.
### Does this PR introduce any user-facing change?
'No'.
### How was this patch tested?
Jenkins test
Closes #28035 from beliefer/supplement-configuration-version.
Authored-by: beliefer <be...@163.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
docs/configuration.md | 88 ++++++++++++++++++++++++++++++++++++++++-----------
1 file changed, 69 insertions(+), 19 deletions(-)
diff --git a/docs/configuration.md b/docs/configuration.md
index 6d01897..fae3bb4 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -143,6 +143,7 @@ of the most common options to set are:
<td>
The name of your application. This will appear in the UI and in log data.
</td>
+ <td>0.9.0</td>
</tr>
<tr>
<td><code>spark.driver.cores</code></td>
@@ -206,6 +207,7 @@ of the most common options to set are:
<code>spark.driver.resource.{resourceName}.discoveryScript</code>
for the driver to find the resource on startup.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.driver.resource.{resourceName}.discoveryScript</code></td>
@@ -216,6 +218,7 @@ of the most common options to set are:
name and an array of addresses. For a client-submitted driver, discovery script must assign
different resource addresses to this driver comparing to other drivers on the same host.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.driver.resource.{resourceName}.vendor</code></td>
@@ -226,6 +229,7 @@ of the most common options to set are:
the Kubernetes device plugin naming convention. (e.g. For GPUs on Kubernetes
this config would be set to nvidia.com or amd.com)
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.resources.discoveryPlugin</code></td>
@@ -293,6 +297,7 @@ of the most common options to set are:
<code>spark.executor.resource.{resourceName}.discoveryScript</code>
for the executor to find the resource on startup.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.executor.resource.{resourceName}.discoveryScript</code></td>
@@ -302,6 +307,7 @@ of the most common options to set are:
write to STDOUT a JSON string in the format of the ResourceInformation class. This has a
name and an array of addresses.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.executor.resource.{resourceName}.vendor</code></td>
@@ -312,6 +318,7 @@ of the most common options to set are:
the Kubernetes device plugin naming convention. (e.g. For GPUs on Kubernetes
this config would be set to nvidia.com or amd.com)
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.extraListeners</code></td>
@@ -337,6 +344,7 @@ of the most common options to set are:
<em>Note:</em> This will be overridden by SPARK_LOCAL_DIRS (Standalone), MESOS_SANDBOX (Mesos) or
LOCAL_DIRS (YARN) environment variables set by the cluster manager.
</td>
+ <td>0.5.0</td>
</tr>
<tr>
<td><code>spark.logConf</code></td>
@@ -344,6 +352,7 @@ of the most common options to set are:
<td>
Logs the effective SparkConf as INFO when a SparkContext is started.
</td>
+ <td>0.9.0</td>
</tr>
<tr>
<td><code>spark.master</code></td>
@@ -352,6 +361,7 @@ of the most common options to set are:
The cluster manager to connect to. See the list of
<a href="submitting-applications.html#master-urls"> allowed master URL's</a>.
</td>
+ <td>0.9.0</td>
</tr>
<tr>
<td><code>spark.submit.deployMode</code></td>
@@ -467,6 +477,7 @@ Apart from these, the following properties are also available, and may be useful
Instead, please set this through the <code>--driver-java-options</code> command line option or in
your default properties file.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.driver.extraJavaOptions</code></td>
@@ -540,6 +551,7 @@ Apart from these, the following properties are also available, and may be useful
verbose gc logging to a file named for the executor ID of the app in /tmp, pass a 'value' of:
<code>-verbose:gc -Xloggc:/tmp/{{APP_ID}}-{{EXECUTOR_ID}}.gc</code>
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.executor.extraJavaOptions</code></td>
@@ -636,6 +648,7 @@ Apart from these, the following properties are also available, and may be useful
Add the environment variable specified by <code>EnvironmentVariableName</code> to the Executor
process. The user can specify multiple of these to set multiple environment variables.
</td>
+ <td>0.9.0</td>
</tr>
<tr>
<td><code>spark.redaction.regex</code></td>
@@ -659,7 +672,7 @@ Apart from these, the following properties are also available, and may be useful
By default the <code>pyspark.profiler.BasicProfiler</code> will be used, but this can be overridden by
passing a profiler class in as a parameter to the <code>SparkContext</code> constructor.
</td>
- <td></td>
+ <td>1.2.0</td>
</tr>
<tr>
<td><code>spark.python.profile.dump</code></td>
@@ -670,6 +683,7 @@ Apart from these, the following properties are also available, and may be useful
by <code>pstats.Stats()</code>. If this is specified, the profile result will not be displayed
automatically.
</td>
+ <td>1.2.0</td>
</tr>
<tr>
<td><code>spark.python.worker.memory</code></td>
@@ -680,6 +694,7 @@ Apart from these, the following properties are also available, and may be useful
(e.g. <code>512m</code>, <code>2g</code>).
If the memory used during aggregation goes above this amount, it will spill the data into disks.
</td>
+ <td>1.1.0</td>
</tr>
<tr>
<td><code>spark.python.worker.reuse</code></td>
@@ -727,6 +742,7 @@ Apart from these, the following properties are also available, and may be useful
repositories given by the command-line option <code>--repositories</code>. For more details, see
<a href="submitting-applications.html#advanced-dependency-management">Advanced Dependency Management</a>.
</td>
+ <td>1.5.0</td>
</tr>
<tr>
<td><code>spark.jars.excludes</code></td>
@@ -735,6 +751,7 @@ Apart from these, the following properties are also available, and may be useful
Comma-separated list of groupId:artifactId, to exclude while resolving the dependencies
provided in <code>spark.jars.packages</code> to avoid dependency conflicts.
</td>
+ <td>1.5.0</td>
</tr>
<tr>
<td><code>spark.jars.ivy</code></td>
@@ -744,6 +761,7 @@ Apart from these, the following properties are also available, and may be useful
<code>spark.jars.packages</code>. This will override the Ivy property <code>ivy.default.ivy.user.dir</code>
which defaults to ~/.ivy2.
</td>
+ <td>1.3.0</td>
</tr>
<tr>
<td><code>spark.jars.ivySettings</code></td>
@@ -756,6 +774,7 @@ Apart from these, the following properties are also available, and may be useful
artifact server like Artifactory. Details on the settings file format can be
found at <a href="http://ant.apache.org/ivy/history/latest-milestone/settings.html">Settings Files</a>
</td>
+ <td>2.2.0</td>
</tr>
<tr>
<td><code>spark.jars.repositories</code></td>
@@ -764,6 +783,7 @@ Apart from these, the following properties are also available, and may be useful
Comma-separated list of additional remote repositories to search for the maven coordinates
given with <code>--packages</code> or <code>spark.jars.packages</code>.
</td>
+ <td>2.3.0</td>
</tr>
<tr>
<td><code>spark.pyspark.driver.python</code></td>
@@ -849,6 +869,7 @@ Apart from these, the following properties are also available, and may be useful
set to a non-zero value. This retry logic helps stabilize large shuffles in the face of long GC
pauses or transient network connectivity issues.
</td>
+ <td>1.2.0</td>
</tr>
<tr>
<td><code>spark.shuffle.io.numConnectionsPerPeer</code></td>
@@ -858,6 +879,7 @@ Apart from these, the following properties are also available, and may be useful
large clusters. For clusters with many hard disks and few hosts, this may result in insufficient
concurrency to saturate all disks, and so users may consider increasing this value.
</td>
+ <td>1.2.1</td>
</tr>
<tr>
<td><code>spark.shuffle.io.preferDirectBufs</code></td>
@@ -867,6 +889,7 @@ Apart from these, the following properties are also available, and may be useful
block transfer. For environments where off-heap memory is tightly limited, users may wish to
turn this off to force all allocations from Netty to be on-heap.
</td>
+ <td>1.2.0</td>
</tr>
<tr>
<td><code>spark.shuffle.io.retryWait</code></td>
@@ -875,6 +898,7 @@ Apart from these, the following properties are also available, and may be useful
(Netty only) How long to wait between retries of fetches. The maximum delay caused by retrying
is 15 seconds by default, calculated as <code>maxRetries * retryWait</code>.
</td>
+ <td>1.2.1</td>
</tr>
<tr>
<td><code>spark.shuffle.io.backLog</code></td>
@@ -887,6 +911,7 @@ Apart from these, the following properties are also available, and may be useful
application (see <code>spark.shuffle.service.enabled</code> option below). If set below 1,
will fallback to OS default defined by Netty's <code>io.netty.util.NetUtil#SOMAXCONN</code>.
</td>
+ <td>1.1.1</td>
</tr>
<tr>
<td><code>spark.shuffle.service.enabled</code></td>
@@ -915,6 +940,7 @@ Apart from these, the following properties are also available, and may be useful
<td>
Cache entries limited to the specified memory footprint, in bytes unless otherwise specified.
</td>
+ <td>2.3.0</td>
</tr>
<tr>
<td><code>spark.shuffle.maxChunksBeingTransferred</code></td>
@@ -926,6 +952,7 @@ Apart from these, the following properties are also available, and may be useful
<code>spark.shuffle.io.retryWait</code>), if those limits are reached the task will fail with
fetch failure.
</td>
+ <td>2.3.0</td>
</tr>
<tr>
<td><code>spark.shuffle.sort.bypassMergeThreshold</code></td>
@@ -1241,6 +1268,7 @@ Apart from these, the following properties are also available, and may be useful
<td>
How many finished batches the Spark UI and status APIs remember before garbage collecting.
</td>
+ <td>1.0.0</td>
</tr>
<tr>
<td><code>spark.ui.retainedDeadExecutors</code></td>
@@ -1634,6 +1662,7 @@ Apart from these, the following properties are also available, and may be useful
Default number of partitions in RDDs returned by transformations like <code>join</code>,
<code>reduceByKey</code>, and <code>parallelize</code> when not set by user.
</td>
+ <td>0.5.0</td>
</tr>
<tr>
<td><code>spark.executor.heartbeatInterval</code></td>
@@ -1653,6 +1682,7 @@ Apart from these, the following properties are also available, and may be useful
Communication timeout to use when fetching files added through SparkContext.addFile() from
the driver.
</td>
+ <td>1.0.0</td>
</tr>
<tr>
<td><code>spark.files.useFetchCache</code></td>
@@ -1665,6 +1695,7 @@ Apart from these, the following properties are also available, and may be useful
disabled in order to use Spark local directories that reside on NFS filesystems (see
<a href="https://issues.apache.org/jira/browse/SPARK-6313">SPARK-6313</a> for more details).
</td>
+ <td>1.2.2</td>
</tr>
<tr>
<td><code>spark.files.overwrite</code></td>
@@ -1673,6 +1704,7 @@ Apart from these, the following properties are also available, and may be useful
Whether to overwrite files added through SparkContext.addFile() when the target file exists and
its contents do not match those of the source.
</td>
+ <td>1.0.0</td>
</tr>
<tr>
<td><code>spark.files.maxPartitionBytes</code></td>
@@ -1693,23 +1725,29 @@ Apart from these, the following properties are also available, and may be useful
<td>2.1.0</td>
</tr>
<tr>
- <td><code>spark.hadoop.cloneConf</code></td>
- <td>false</td>
- <td>If set to true, clones a new Hadoop <code>Configuration</code> object for each task. This
+ <td><code>spark.hadoop.cloneConf</code></td>
+ <td>false</td>
+ <td>
+ If set to true, clones a new Hadoop <code>Configuration</code> object for each task. This
option should be enabled to work around <code>Configuration</code> thread-safety issues (see
<a href="https://issues.apache.org/jira/browse/SPARK-2546">SPARK-2546</a> for more details).
This is disabled by default in order to avoid unexpected performance regressions for jobs that
- are not affected by these issues.</td>
+ are not affected by these issues.
+ </td>
+ <td>1.0.3</td>
</tr>
<tr>
- <td><code>spark.hadoop.validateOutputSpecs</code></td>
- <td>true</td>
- <td>If set to true, validates the output specification (e.g. checking if the output directory already exists)
+ <td><code>spark.hadoop.validateOutputSpecs</code></td>
+ <td>true</td>
+ <td>
+ If set to true, validates the output specification (e.g. checking if the output directory already exists)
used in saveAsHadoopFile and other variants. This can be disabled to silence exceptions due to pre-existing
- output directories. We recommend that users do not disable this except if trying to achieve compatibility with
- previous versions of Spark. Simply use Hadoop's FileSystem API to delete output directories by hand.
- This setting is ignored for jobs generated through Spark Streaming's StreamingContext, since
- data may need to be rewritten to pre-existing output directories during checkpoint recovery.</td>
+ output directories. We recommend that users do not disable this except if trying to achieve compatibility
+ with previous versions of Spark. Simply use Hadoop's FileSystem API to delete output directories by hand.
+ This setting is ignored for jobs generated through Spark Streaming's StreamingContext, since data may
+ need to be rewritten to pre-existing output directories during checkpoint recovery.
+ </td>
+ <td>1.0.1</td>
</tr>
<tr>
<td><code>spark.storage.memoryMapThreshold</code></td>
@@ -1729,6 +1767,7 @@ Apart from these, the following properties are also available, and may be useful
Version 2 may have better performance, but version 1 may handle failures better in certain situations,
as per <a href="https://issues.apache.org/jira/browse/MAPREDUCE-4815">MAPREDUCE-4815</a>.
</td>
+ <td>2.2.0</td>
</tr>
</table>
@@ -1843,7 +1882,7 @@ Apart from these, the following properties are also available, and may be useful
need to be increased, so that incoming connections are not dropped when a large number of
connections arrives in a short period of time.
</td>
- <td></td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.network.timeout</code></td>
@@ -1866,7 +1905,7 @@ Apart from these, the following properties are also available, and may be useful
block transfer. For environments where off-heap memory is tightly limited, users may wish to
turn this off to force all allocations to be on-heap.
</td>
- <td></td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.port.maxRetries</code></td>
@@ -1878,7 +1917,7 @@ Apart from these, the following properties are also available, and may be useful
essentially allows it to try a range of ports from the start port specified
to port + maxRetries.
</td>
- <td></td>
+ <td>1.1.1</td>
</tr>
<tr>
<td><code>spark.rpc.numRetries</code></td>
@@ -1921,7 +1960,7 @@ Apart from these, the following properties are also available, and may be useful
out and giving up. To avoid unwilling timeout caused by long pause like GC,
you can set larger value.
</td>
- <td></td>
+ <td>1.1.1</td>
</tr>
<tr>
<td><code>spark.network.maxRemoteBlockSizeFetchToMem</code></td>
@@ -2054,6 +2093,7 @@ Apart from these, the following properties are also available, and may be useful
that register to the listener bus. Consider increasing value, if the listener events corresponding
to shared queue are dropped. Increasing this value may result in the driver using more memory.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.scheduler.listenerbus.eventqueue.appStatus.capacity</code></td>
@@ -2063,6 +2103,7 @@ Apart from these, the following properties are also available, and may be useful
Consider increasing value, if the listener events corresponding to appStatus queue are dropped.
Increasing this value may result in the driver using more memory.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.scheduler.listenerbus.eventqueue.executorManagement.capacity</code></td>
@@ -2072,6 +2113,7 @@ Apart from these, the following properties are also available, and may be useful
executor management listeners. Consider increasing value if the listener events corresponding to
executorManagement queue are dropped. Increasing this value may result in the driver using more memory.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.scheduler.listenerbus.eventqueue.eventLog.capacity</code></td>
@@ -2081,6 +2123,7 @@ Apart from these, the following properties are also available, and may be useful
that write events to eventLogs. Consider increasing value if the listener events corresponding to eventLog queue
are dropped. Increasing this value may result in the driver using more memory.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.scheduler.listenerbus.eventqueue.streams.capacity</code></td>
@@ -2090,6 +2133,7 @@ Apart from these, the following properties are also available, and may be useful
Consider increasing value if the listener events corresponding to streams queue are dropped. Increasing
this value may result in the driver using more memory.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.scheduler.blacklist.unschedulableTaskSetTimeout</code></td>
@@ -2272,6 +2316,7 @@ Apart from these, the following properties are also available, and may be useful
in order to assign resource slots (e.g. a 0.2222 configuration, or 1/0.2222 slots will become
4 tasks/resource, not 5).
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.task.maxFailures</code></td>
@@ -2336,6 +2381,7 @@ Apart from these, the following properties are also available, and may be useful
<td>
Number of consecutive stage attempts allowed before a stage is aborted.
</td>
+ <td>2.2.0</td>
</tr>
</table>
@@ -2528,13 +2574,14 @@ like shuffle, just replace "rpc" with "shuffle" in the property names except
<code>spark.{driver|executor}.rpc.netty.dispatcher.numThreads</code>, which is only for RPC module.
<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.{driver|executor}.rpc.io.serverThreads</code></td>
<td>
Fall back on <code>spark.rpc.io.serverThreads</code>
</td>
<td>Number of threads used in the server thread pool</td>
+ <td>1.6.0</td>
</tr>
<tr>
<td><code>spark.{driver|executor}.rpc.io.clientThreads</code></td>
@@ -2542,6 +2589,7 @@ like shuffle, just replace "rpc" with "shuffle" in the property names except
Fall back on <code>spark.rpc.io.clientThreads</code>
</td>
<td>Number of threads used in the client thread pool</td>
+ <td>1.6.0</td>
</tr>
<tr>
<td><code>spark.{driver|executor}.rpc.netty.dispatcher.numThreads</code></td>
@@ -2549,6 +2597,7 @@ like shuffle, just replace "rpc" with "shuffle" in the property names except
Fall back on <code>spark.rpc.netty.dispatcher.numThreads</code>
</td>
<td>Number of threads used in RPC message dispatcher thread pool</td>
+ <td>3.0.0</td>
</tr>
</table>
@@ -2730,7 +2779,7 @@ Spark subsystems.
<td>
Executable for executing R scripts in client modes for driver. Ignored in cluster modes.
</td>
- <td></td>
+ <td>1.5.3</td>
</tr>
<tr>
<td><code>spark.r.shell.command</code></td>
@@ -2739,7 +2788,7 @@ Spark subsystems.
Executable for executing sparkR shell in client modes for driver. Ignored in cluster modes. It is the same as environment variable <code>SPARKR_DRIVER_R</code>, but take precedence over it.
<code>spark.r.shell.command</code> is used for sparkR shell while <code>spark.r.driver.command</code> is used for running R script.
</td>
- <td></td>
+ <td>2.1.0</td>
</tr>
<tr>
<td><code>spark.r.backendConnectionTimeout</code></td>
@@ -2771,6 +2820,7 @@ Spark subsystems.
Checkpoint interval for graph and message in Pregel. It used to avoid stackOverflowError due to long lineage chains
after lots of iterations. The checkpoint is disabled by default.
</td>
+ <td>2.2.0</td>
</tr>
</table>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 02/04: [SPARK-31295][DOC][FOLLOWUP] Supplement version for
configuration appear in doc
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 8af58ebd958813e9ff29d2f0d1b070d529ba1275
Author: beliefer <be...@163.com>
AuthorDate: Thu Apr 2 16:01:54 2020 +0900
[SPARK-31295][DOC][FOLLOWUP] Supplement version for configuration appear in doc
### What changes were proposed in this pull request?
This PR supplements version for configuration appear in docs.
I sorted out some information show below.
**docs/sql-performance-tuning.md**
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.sql.inMemoryColumnarStorage.compressed | 1.0.1 | SPARK-2631 | 86534d0f5255362618c05a07b0171ec35c915822#diff-41ef65b9ef5b518f77e2a03559893f4d |
spark.sql.inMemoryColumnarStorage.batchSize | 1.1.1 | SPARK-2650 | 779d1eb26d0f031791e93c908d51a59c3b422a55#diff-41ef65b9ef5b518f77e2a03559893f4d |
spark.sql.files.maxPartitionBytes | 2.0.0 | SPARK-13664 | 17eec0a71ba8713c559d641e3f43a1be726b037c#diff-32bb9518401c0948c5ea19377b5069ab |
spark.sql.files.openCostInBytes | 2.0.0 | SPARK-14259 | 400b2f863ffaa01a34a8dae1541c61526fef908b#diff-32bb9518401c0948c5ea19377b5069ab |
spark.sql.broadcastTimeout | 1.3.0 | SPARK-4269 | fa66ef6c97e87c9255b67b03836a4ba50598ebae#diff-41ef65b9ef5b518f77e2a03559893f4d |
spark.sql.autoBroadcastJoinThreshold | 1.1.0 | SPARK-2393 | c7db274be79f448fda566208946cb50958ea9b1a#diff-41ef65b9ef5b518f77e2a03559893f4d |
spark.sql.shuffle.partitions | 1.1.0 | SPARK-1508 | 08ed9ad81397b71206c4dc903bfb94b6105691ed#diff-41ef65b9ef5b518f77e2a03559893f4d |
spark.sql.adaptive.coalescePartitions.enabled | 3.0.0 | SPARK-31037 | 46b7f1796bd0b96977ce9b473601033f397a3b18#diff-9a6b543db706f1a90f790783d6930a13 |
spark.sql.adaptive.coalescePartitions.minPartitionNum | 3.0.0 | SPARK-31037 | 46b7f1796bd0b96977ce9b473601033f397a3b18#diff-9a6b543db706f1a90f790783d6930a13 |
spark.sql.adaptive.coalescePartitions.initialPartitionNum | 3.0.0 | SPARK-31037 | 46b7f1796bd0b96977ce9b473601033f397a3b18#diff-9a6b543db706f1a90f790783d6930a13 |
spark.sql.adaptive.advisoryPartitionSizeInBytes | 3.0.0 | SPARK-31037 | 46b7f1796bd0b96977ce9b473601033f397a3b18#diff-9a6b543db706f1a90f790783d6930a13 |
spark.sql.adaptive.skewJoin.enabled | 3.0.0 | SPARK-31037 | 46b7f1796bd0b96977ce9b473601033f397a3b18#diff-9a6b543db706f1a90f790783d6930a13 |
spark.sql.adaptive.skewJoin.skewedPartitionFactor | 3.0.0 | SPARK-31037 | 46b7f1796bd0b96977ce9b473601033f397a3b18#diff-9a6b543db706f1a90f790783d6930a13 |
spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes | 3.0.0 | SPARK-31201 | 8d0800a0803d3c47938bddefa15328d654739bc5#diff-9a6b543db706f1a90f790783d6930a13 |
**docs/sql-ref-ansi-compliance.md**
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.sql.ansi.enabled | 3.0.0 | SPARK-30125 | d9b30694122f8716d3acb448638ef1e2b96ebc7a#diff-9a6b543db706f1a90f790783d6930a13 |
spark.sql.storeAssignmentPolicy | 3.0.0 | SPARK-28730 | 895c90b582cc2b2667241f66d5b733852aeef9eb#diff-9a6b543db706f1a90f790783d6930a13 |
### Why are the changes needed?
Supplemental configuration version information.
### Does this PR introduce any user-facing change?
'No'.
### How was this patch tested?
Jenkins test
Closes #28096 from beliefer/supplement-version-of-performance.
Authored-by: beliefer <be...@163.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
docs/sql-performance-tuning.md | 30 ++++++++++++++++++++++--------
docs/sql-ref-ansi-compliance.md | 4 +++-
2 files changed, 25 insertions(+), 9 deletions(-)
diff --git a/docs/sql-performance-tuning.md b/docs/sql-performance-tuning.md
index 9a1cc89..279aad6 100644
--- a/docs/sql-performance-tuning.md
+++ b/docs/sql-performance-tuning.md
@@ -35,7 +35,7 @@ Configuration of in-memory caching can be done using the `setConf` method on `Sp
`SET key=value` commands using SQL.
<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.sql.inMemoryColumnarStorage.compressed</code></td>
<td>true</td>
@@ -43,6 +43,7 @@ Configuration of in-memory caching can be done using the `setConf` method on `Sp
When set to true Spark SQL will automatically select a compression codec for each column based
on statistics of the data.
</td>
+ <td>1.0.1</td>
</tr>
<tr>
<td><code>spark.sql.inMemoryColumnarStorage.batchSize</code></td>
@@ -51,6 +52,7 @@ Configuration of in-memory caching can be done using the `setConf` method on `Sp
Controls the size of batches for columnar caching. Larger batch sizes can improve memory utilization
and compression, but risk OOMs when caching data.
</td>
+ <td>1.1.1</td>
</tr>
</table>
@@ -61,7 +63,7 @@ The following options can also be used to tune the performance of query executio
that these options will be deprecated in future release as more optimizations are performed automatically.
<table class="table">
- <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+ <tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.sql.files.maxPartitionBytes</code></td>
<td>134217728 (128 MB)</td>
@@ -69,6 +71,7 @@ that these options will be deprecated in future release as more optimizations ar
The maximum number of bytes to pack into a single partition when reading files.
This configuration is effective only when using file-based sources such as Parquet, JSON and ORC.
</td>
+ <td>2.0.0</td>
</tr>
<tr>
<td><code>spark.sql.files.openCostInBytes</code></td>
@@ -80,15 +83,17 @@ that these options will be deprecated in future release as more optimizations ar
scheduled first). This configuration is effective only when using file-based sources such as Parquet,
JSON and ORC.
</td>
+ <td>2.0.0</td>
</tr>
<tr>
<td><code>spark.sql.broadcastTimeout</code></td>
<td>300</td>
<td>
- <p>
- Timeout in seconds for the broadcast wait time in broadcast joins
- </p>
+ <p>
+ Timeout in seconds for the broadcast wait time in broadcast joins
+ </p>
</td>
+ <td>1.3.0</td>
</tr>
<tr>
<td><code>spark.sql.autoBroadcastJoinThreshold</code></td>
@@ -99,6 +104,7 @@ that these options will be deprecated in future release as more optimizations ar
statistics are only supported for Hive Metastore tables where the command
<code>ANALYZE TABLE <tableName> COMPUTE STATISTICS noscan</code> has been run.
</td>
+ <td>1.1.0</td>
</tr>
<tr>
<td><code>spark.sql.shuffle.partitions</code></td>
@@ -106,6 +112,7 @@ that these options will be deprecated in future release as more optimizations ar
<td>
Configures the number of partitions to use when shuffling data for joins or aggregations.
</td>
+ <td>1.1.0</td>
</tr>
</table>
@@ -193,13 +200,14 @@ Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that ma
### Coalescing Post Shuffle Partitions
This feature coalesces the post shuffle partitions based on the map output statistics when both `spark.sql.adaptive.enabled` and `spark.sql.adaptive.coalescePartitions.enabled` configurations are true. This feature simplifies the tuning of shuffle partition number when running queries. You do not need to set a proper shuffle partition number to fit your dataset. Spark can pick the proper shuffle partition number at runtime once you set a large enough initial number of shuffle partitions [...]
<table class="table">
- <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+ <tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.sql.adaptive.coalescePartitions.enabled</code></td>
<td>true</td>
<td>
When true and <code>spark.sql.adaptive.enabled</code> is true, Spark will coalesce contiguous shuffle partitions according to the target size (specified by <code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code>), to avoid too many small tasks.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.sql.adaptive.coalescePartitions.minPartitionNum</code></td>
@@ -207,6 +215,7 @@ This feature coalesces the post shuffle partitions based on the map output stati
<td>
The minimum number of shuffle partitions after coalescing. If not set, the default value is the default parallelism of the Spark cluster. This configuration only has an effect when <code>spark.sql.adaptive.enabled</code> and <code>spark.sql.adaptive.coalescePartitions.enabled</code> are both enabled.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.sql.adaptive.coalescePartitions.initialPartitionNum</code></td>
@@ -214,6 +223,7 @@ This feature coalesces the post shuffle partitions based on the map output stati
<td>
The initial number of shuffle partitions before coalescing. By default it equals to <code>spark.sql.shuffle.partitions</code>. This configuration only has an effect when <code>spark.sql.adaptive.enabled</code> and <code>spark.sql.adaptive.coalescePartitions.enabled</code> are both enabled.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code></td>
@@ -221,6 +231,7 @@ This feature coalesces the post shuffle partitions based on the map output stati
<td>
The advisory size in bytes of the shuffle partition during adaptive optimization (when <code>spark.sql.adaptive.enabled</code> is true). It takes effect when Spark coalesces small shuffle partitions or splits skewed shuffle partition.
</td>
+ <td>3.0.0</td>
</tr>
</table>
@@ -230,13 +241,14 @@ AQE converts sort-merge join to broadcast hash join when the runtime statistics
### Optimizing Skew Join
Data skew can severely downgrade the performance of join queries. This feature dynamically handles skew in sort-merge join by splitting (and replicating if needed) skewed tasks into roughly evenly sized tasks. It takes effect when both `spark.sql.adaptive.enabled` and `spark.sql.adaptive.skewJoin.enabled` configurations are enabled.
<table class="table">
- <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+ <tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.sql.adaptive.skewJoin.enabled</code></td>
<td>true</td>
<td>
When true and <code>spark.sql.adaptive.enabled</code> is true, Spark dynamically handles skew in sort-merge join by splitting (and replicating if needed) skewed partitions.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.sql.adaptive.skewJoin.skewedPartitionFactor</code></td>
@@ -244,6 +256,7 @@ Data skew can severely downgrade the performance of join queries. This feature d
<td>
A partition is considered as skewed if its size is larger than this factor multiplying the median partition size and also larger than <code>spark.sql.adaptive.skewedPartitionThresholdInBytes</code>.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes</code></td>
@@ -251,5 +264,6 @@ Data skew can severely downgrade the performance of join queries. This feature d
<td>
A partition is considered as skewed if its size in bytes is larger than this threshold and also larger than <code>spark.sql.adaptive.skewJoin.skewedPartitionFactor</code> multiplying the median partition size. Ideally this config should be set larger than <code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code>.
</td>
+ <td>3.0.0</td>
</tr>
- </table>
\ No newline at end of file
+ </table>
diff --git a/docs/sql-ref-ansi-compliance.md b/docs/sql-ref-ansi-compliance.md
index bc5bde6..83affb9 100644
--- a/docs/sql-ref-ansi-compliance.md
+++ b/docs/sql-ref-ansi-compliance.md
@@ -28,7 +28,7 @@ The casting behaviours are defined as store assignment rules in the standard.
When `spark.sql.storeAssignmentPolicy` is set to `ANSI`, Spark SQL complies with the ANSI store assignment rules. This is a separate configuration because its default value is `ANSI`, while the configuration `spark.sql.ansi.enabled` is disabled by default.
<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.sql.ansi.enabled</code></td>
<td>false</td>
@@ -37,6 +37,7 @@ When `spark.sql.storeAssignmentPolicy` is set to `ANSI`, Spark SQL complies with
1. Spark will throw a runtime exception if an overflow occurs in any operation on integral/decimal field.
2. Spark will forbid using the reserved keywords of ANSI SQL as identifiers in the SQL parser.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.sql.storeAssignmentPolicy</code></td>
@@ -52,6 +53,7 @@ When `spark.sql.storeAssignmentPolicy` is set to `ANSI`, Spark SQL complies with
With strict policy, Spark doesn't allow any possible precision loss or data truncation in type coercion,
e.g. converting double to int or decimal to double is not allowed.
</td>
+ <td>3.0.0</td>
</tr>
</table>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 01/04: [SPARK-31295][DOC] Supplement version for
configuration appear in doc
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit dec53bedbd4acbde0b7a6064476dbd148e4c6d08
Author: beliefer <be...@163.com>
AuthorDate: Tue Mar 31 12:33:46 2020 +0900
[SPARK-31295][DOC] Supplement version for configuration appear in doc
### What changes were proposed in this pull request?
This PR supplements version for configuration appear in docs.
I sorted out some information show below.
**docs/spark-standalone.md**
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.deploy.retainedApplications | 0.8.0 | None | 46eecd110a4017ea0c86cbb1010d0ccd6a5eb2ef#diff-29dffdccd5a7f4c8b496c293e87c8668 |
spark.deploy.retainedDrivers | 1.1.0 | None | 7446f5ff93142d2dd5c79c63fa947f47a1d4db8b#diff-29dffdccd5a7f4c8b496c293e87c8668 |
spark.deploy.spreadOut | 0.6.1 | None | bb2b9ff37cd2503cc6ea82c5dd395187b0910af0#diff-0e7ae91819fc8f7b47b0f97be7116325 |
spark.deploy.defaultCores | 0.9.0 | None | d8bcc8e9a095c1b20dd7a17b6535800d39bff80e#diff-29dffdccd5a7f4c8b496c293e87c8668 |
spark.deploy.maxExecutorRetries | 1.6.3 | SPARK-16956 | ace458f0330f22463ecf7cbee7c0465e10fba8a8#diff-29dffdccd5a7f4c8b496c293e87c8668 |
spark.worker.resource.{resourceName}.amount | 3.0.0 | SPARK-27371 | cbad616d4cb0c58993a88df14b5e30778c7f7e85#diff-d25032e4a3ae1b85a59e4ca9ccf189a8 |
spark.worker.resource.{resourceName}.discoveryScript | 3.0.0 | SPARK-27371 | cbad616d4cb0c58993a88df14b5e30778c7f7e85#diff-d25032e4a3ae1b85a59e4ca9ccf189a8 |
spark.worker.resourcesFile | 3.0.0 | SPARK-27369 | 7cbe01e8efc3f6cd3a0cac4bcfadea8fcc74a955#diff-b2fc8d6ab7ac5735085e2d6cfacb95da |
spark.shuffle.service.db.enabled | 3.0.0 | SPARK-26288 | 8b0aa59218c209d39cbba5959302d8668b885cf6#diff-6bdad48cfc34314e89599655442ff210 |
spark.storage.cleanupFilesAfterExecutorExit | 2.4.0 | SPARK-24340 | 8ef167a5f9ba8a79bb7ca98a9844fe9cfcfea060#diff-916ca56b663f178f302c265b7ef38499 |
spark.deploy.recoveryMode | 0.8.1 | None | d66c01f2b6defb3db6c1be99523b734a4d960532#diff-29dffdccd5a7f4c8b496c293e87c8668 |
spark.deploy.recoveryDirectory | 0.8.1 | None | d66c01f2b6defb3db6c1be99523b734a4d960532#diff-29dffdccd5a7f4c8b496c293e87c8668 |
**docs/sql-data-sources-avro.md**
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.sql.legacy.replaceDatabricksSparkAvro.enabled | 2.4.0 | SPARK-25129 | ac0174e55af2e935d41545721e9f430c942b3a0c#diff-9a6b543db706f1a90f790783d6930a13 |
spark.sql.avro.compression.codec | 2.4.0 | SPARK-24881 | 0a0f68bae6c0a1bf30184b1e9ac6bf3805bd7511#diff-9a6b543db706f1a90f790783d6930a13 |
spark.sql.avro.deflate.level | 2.4.0 | SPARK-24881 | 0a0f68bae6c0a1bf30184b1e9ac6bf3805bd7511#diff-9a6b543db706f1a90f790783d6930a13 |
**docs/sql-data-sources-orc.md**
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.sql.orc.impl | 2.3.0 | SPARK-20728 | 326f1d6728a7734c228d8bfaa69442a1c7b92e9b#diff-9a6b543db706f1a90f790783d6930a13 |
spark.sql.orc.enableVectorizedReader | 2.3.0 | SPARK-16060 | 60f6b994505e3f82091a04eed2dc0a9e8bd523ce#diff-9a6b543db706f1a90f790783d6930a13 |
**docs/sql-data-sources-parquet.md**
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.sql.parquet.binaryAsString | 1.1.1 | SPARK-2927 | de501e169f24e4573747aec85b7651c98633c028#diff-41ef65b9ef5b518f77e2a03559893f4d |
spark.sql.parquet.int96AsTimestamp | 1.3.0 | SPARK-4987 | 67d52207b5cf2df37ca70daff2a160117510f55e#diff-41ef65b9ef5b518f77e2a03559893f4d |
spark.sql.parquet.compression.codec | 1.1.1 | SPARK-3131 | 3a9d874d7a46ab8b015631d91ba479d9a0ba827f#diff-41ef65b9ef5b518f77e2a03559893f4d |
spark.sql.parquet.filterPushdown | 1.2.0 | SPARK-4391 | 576688aa2a19bd4ba239a2b93af7947f983e5124#diff-41ef65b9ef5b518f77e2a03559893f4d |
spark.sql.hive.convertMetastoreParquet | 1.1.1 | SPARK-2406 | cc4015d2fa3785b92e6ab079b3abcf17627f7c56#diff-ff50aea397a607b79df9bec6f2a841db |
spark.sql.parquet.mergeSchema | 1.5.0 | SPARK-8690 | 246265f2bb056d5e9011d3331b809471a24ff8d7#diff-41ef65b9ef5b518f77e2a03559893f4d |
spark.sql.parquet.writeLegacyFormat | 1.6.0 | SPARK-10400 | 01cd688f5245cbb752863100b399b525b31c3510#diff-41ef65b9ef5b518f77e2a03559893f4d |
### Why are the changes needed?
Supplemental configuration version information.
### Does this PR introduce any user-facing change?
'No'.
### How was this patch tested?
Jenkins test
Closes #28064 from beliefer/supplement-doc-for-data-sources.
Authored-by: beliefer <be...@163.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
docs/spark-standalone.md | 13 ++++++++++++-
docs/sql-data-sources-avro.md | 21 +++++++++++++++++----
docs/sql-data-sources-orc.md | 16 +++++++++++++---
docs/sql-data-sources-parquet.md | 9 ++++++++-
4 files changed, 50 insertions(+), 9 deletions(-)
diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md
index 4d4b85e..2c2ed53 100644
--- a/docs/spark-standalone.md
+++ b/docs/spark-standalone.md
@@ -192,6 +192,7 @@ SPARK_MASTER_OPTS supports the following system properties:
<td>
The maximum number of completed applications to display. Older applications will be dropped from the UI to maintain this limit.<br/>
</td>
+ <td>0.8.0</td>
</tr>
<tr>
<td><code>spark.deploy.retainedDrivers</code></td>
@@ -199,6 +200,7 @@ SPARK_MASTER_OPTS supports the following system properties:
<td>
The maximum number of completed drivers to display. Older drivers will be dropped from the UI to maintain this limit.<br/>
</td>
+ <td>1.1.0</td>
</tr>
<tr>
<td><code>spark.deploy.spreadOut</code></td>
@@ -208,6 +210,7 @@ SPARK_MASTER_OPTS supports the following system properties:
to consolidate them onto as few nodes as possible. Spreading out is usually better for
data locality in HDFS, but consolidating is more efficient for compute-intensive workloads. <br/>
</td>
+ <td>0.6.1</td>
</tr>
<tr>
<td><code>spark.deploy.defaultCores</code></td>
@@ -219,6 +222,7 @@ SPARK_MASTER_OPTS supports the following system properties:
Set this lower on a shared cluster to prevent users from grabbing
the whole cluster by default. <br/>
</td>
+ <td>0.9.0</td>
</tr>
<tr>
<td><code>spark.deploy.maxExecutorRetries</code></td>
@@ -234,6 +238,7 @@ SPARK_MASTER_OPTS supports the following system properties:
<code>-1</code>.
<br/>
</td>
+ <td>1.6.3</td>
</tr>
<tr>
<td><code>spark.worker.timeout</code></td>
@@ -250,6 +255,7 @@ SPARK_MASTER_OPTS supports the following system properties:
<td>
Amount of a particular resource to use on the worker.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.worker.resource.{resourceName}.discoveryScript</code></td>
@@ -258,6 +264,7 @@ SPARK_MASTER_OPTS supports the following system properties:
Path to resource discovery script, which is used to find a particular resource while worker starting up.
And the output of the script should be formatted like the <code>ResourceInformation</code> class.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.worker.resourcesFile</code></td>
@@ -317,6 +324,7 @@ SPARK_WORKER_OPTS supports the following system properties:
enabled). You should also enable <code>spark.worker.cleanup.enabled</code>, to ensure that the state
eventually gets cleaned up. This config may be removed in the future.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.storage.cleanupFilesAfterExecutorExit</code></td>
@@ -329,6 +337,7 @@ SPARK_WORKER_OPTS supports the following system properties:
all files/subdirectories of a stopped and timeout application.
This only affects Standalone mode, support of other cluster manangers can be added in the future.
</td>
+ <td>2.4.0</td>
</tr>
<tr>
<td><code>spark.worker.ui.compressedLogFileLengthCacheSize</code></td>
@@ -490,14 +499,16 @@ ZooKeeper is the best way to go for production-level high availability, but if y
In order to enable this recovery mode, you can set SPARK_DAEMON_JAVA_OPTS in spark-env using this configuration:
<table class="table">
- <tr><th style="width:21%">System property</th><th>Meaning</th></tr>
+ <tr><th style="width:21%">System property</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.deploy.recoveryMode</code></td>
<td>Set to FILESYSTEM to enable single-node recovery mode (default: NONE).</td>
+ <td>0.8.1</td>
</tr>
<tr>
<td><code>spark.deploy.recoveryDirectory</code></td>
<td>The directory in which Spark will store recovery state, accessible from the Master's perspective.</td>
+ <td>0.8.1</td>
</tr>
</table>
diff --git a/docs/sql-data-sources-avro.md b/docs/sql-data-sources-avro.md
index 8e6a407..d926ae7 100644
--- a/docs/sql-data-sources-avro.md
+++ b/docs/sql-data-sources-avro.md
@@ -258,21 +258,34 @@ Data source options of Avro can be set via:
## Configuration
Configuration of Avro can be done using the `setConf` method on SparkSession or by running `SET key=value` commands using SQL.
<table class="table">
- <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th></tr>
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Since Version</b></th></tr>
<tr>
<td>spark.sql.legacy.replaceDatabricksSparkAvro.enabled</td>
<td>true</td>
- <td>If it is set to true, the data source provider <code>com.databricks.spark.avro</code> is mapped to the built-in but external Avro data source module for backward compatibility.</td>
+ <td>
+ If it is set to true, the data source provider <code>com.databricks.spark.avro</code> is mapped
+ to the built-in but external Avro data source module for backward compatibility.
+ </td>
+ <td>2.4.0</td>
</tr>
<tr>
<td>spark.sql.avro.compression.codec</td>
<td>snappy</td>
- <td>Compression codec used in writing of AVRO files. Supported codecs: uncompressed, deflate, snappy, bzip2 and xz. Default codec is snappy.</td>
+ <td>
+ Compression codec used in writing of AVRO files. Supported codecs: uncompressed, deflate,
+ snappy, bzip2 and xz. Default codec is snappy.
+ </td>
+ <td>2.4.0</td>
</tr>
<tr>
<td>spark.sql.avro.deflate.level</td>
<td>-1</td>
- <td>Compression level for the deflate codec used in writing of AVRO files. Valid value must be in the range of from 1 to 9 inclusive or -1. The default value is -1 which corresponds to 6 level in the current implementation.</td>
+ <td>
+ Compression level for the deflate codec used in writing of AVRO files. Valid value must be in
+ the range of from 1 to 9 inclusive or -1. The default value is -1 which corresponds to 6 level
+ in the current implementation.
+ </td>
+ <td>2.4.0</td>
</tr>
</table>
diff --git a/docs/sql-data-sources-orc.md b/docs/sql-data-sources-orc.md
index bddffe0..4c4b3b1 100644
--- a/docs/sql-data-sources-orc.md
+++ b/docs/sql-data-sources-orc.md
@@ -27,15 +27,25 @@ serde tables (e.g., the ones created using the clause `USING HIVE OPTIONS (fileF
the vectorized reader is used when `spark.sql.hive.convertMetastoreOrc` is also set to `true`.
<table class="table">
- <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th></tr>
+ <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Since Version</b></th></tr>
<tr>
<td><code>spark.sql.orc.impl</code></td>
<td><code>native</code></td>
- <td>The name of ORC implementation. It can be one of <code>native</code> and <code>hive</code>. <code>native</code> means the native ORC support. <code>hive</code> means the ORC library in Hive.</td>
+ <td>
+ The name of ORC implementation. It can be one of <code>native</code> and <code>hive</code>.
+ <code>native</code> means the native ORC support. <code>hive</code> means the ORC library
+ in Hive.
+ </td>
+ <td>2.3.0</td>
</tr>
<tr>
<td><code>spark.sql.orc.enableVectorizedReader</code></td>
<td><code>true</code></td>
- <td>Enables vectorized orc decoding in <code>native</code> implementation. If <code>false</code>, a new non-vectorized ORC reader is used in <code>native</code> implementation. For <code>hive</code> implementation, this is ignored.</td>
+ <td>
+ Enables vectorized orc decoding in <code>native</code> implementation. If <code>false</code>,
+ a new non-vectorized ORC reader is used in <code>native</code> implementation.
+ For <code>hive</code> implementation, this is ignored.
+ </td>
+ <td>2.3.0</td>
</tr>
</table>
diff --git a/docs/sql-data-sources-parquet.md b/docs/sql-data-sources-parquet.md
index 53a1111..6e52446 100644
--- a/docs/sql-data-sources-parquet.md
+++ b/docs/sql-data-sources-parquet.md
@@ -258,7 +258,7 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession
`SET key=value` commands using SQL.
<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.sql.parquet.binaryAsString</code></td>
<td>false</td>
@@ -267,6 +267,7 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession
not differentiate between binary data and strings when writing out the Parquet schema. This
flag tells Spark SQL to interpret binary data as a string to provide compatibility with these systems.
</td>
+ <td>1.1.1</td>
</tr>
<tr>
<td><code>spark.sql.parquet.int96AsTimestamp</code></td>
@@ -275,6 +276,7 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession
Some Parquet-producing systems, in particular Impala and Hive, store Timestamp into INT96. This
flag tells Spark SQL to interpret INT96 data as a timestamp to provide compatibility with these systems.
</td>
+ <td>1.3.0</td>
</tr>
<tr>
<td><code>spark.sql.parquet.compression.codec</code></td>
@@ -287,11 +289,13 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession
Note that <code>zstd</code> requires <code>ZStandardCodec</code> to be installed before Hadoop 2.9.0, <code>brotli</code> requires
<code>BrotliCodec</code> to be installed.
</td>
+ <td>1.1.1</td>
</tr>
<tr>
<td><code>spark.sql.parquet.filterPushdown</code></td>
<td>true</td>
<td>Enables Parquet filter push-down optimization when set to true.</td>
+ <td>1.2.0</td>
</tr>
<tr>
<td><code>spark.sql.hive.convertMetastoreParquet</code></td>
@@ -300,6 +304,7 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession
When set to false, Spark SQL will use the Hive SerDe for parquet tables instead of the built in
support.
</td>
+ <td>1.1.1</td>
</tr>
<tr>
<td><code>spark.sql.parquet.mergeSchema</code></td>
@@ -310,6 +315,7 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession
schema is picked from the summary file or a random data file if no summary file is available.
</p>
</td>
+ <td>1.5.0</td>
</tr>
<tr>
<td><code>spark.sql.parquet.writeLegacyFormat</code></td>
@@ -321,5 +327,6 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession
example, decimals will be written in int-based format. If Parquet output is intended for use
with systems that do not support this newer format, set to true.
</td>
+ <td>1.6.0</td>
</tr>
</table>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org