You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2018/05/02 02:55:11 UTC
spark git commit: [SPARK-24131][PYSPARK] Add majorMinorVersion API to
PySpark for determining Spark versions
Repository: spark
Updated Branches:
refs/heads/master 6782359a0 -> e15850be6
[SPARK-24131][PYSPARK] Add majorMinorVersion API to PySpark for determining Spark versions
## What changes were proposed in this pull request?
We need to determine Spark major and minor versions in PySpark. We can add a `majorMinorVersion` API to PySpark which is similar to the Scala API in `VersionUtils.majorMinorVersion`.
## How was this patch tested?
Added tests.
Author: Liang-Chi Hsieh <vi...@gmail.com>
Closes #21203 from viirya/SPARK-24131.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e15850be
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e15850be
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e15850be
Branch: refs/heads/master
Commit: e15850be6e0210614a734a307f5b83bdf44e2456
Parents: 6782359
Author: Liang-Chi Hsieh <vi...@gmail.com>
Authored: Wed May 2 10:55:01 2018 +0800
Committer: hyukjinkwon <gu...@apache.org>
Committed: Wed May 2 10:55:01 2018 +0800
----------------------------------------------------------------------
python/pyspark/util.py | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/e15850be/python/pyspark/util.py
----------------------------------------------------------------------
diff --git a/python/pyspark/util.py b/python/pyspark/util.py
index 49afc13..04df835 100644
--- a/python/pyspark/util.py
+++ b/python/pyspark/util.py
@@ -16,6 +16,7 @@
# limitations under the License.
#
+import re
import sys
import inspect
from py4j.protocol import Py4JJavaError
@@ -61,6 +62,26 @@ def _get_argspec(f):
return argspec
+def majorMinorVersion(version):
+ """
+ Get major and minor version numbers for given Spark version string.
+
+ >>> version = "2.4.0"
+ >>> majorMinorVersion(version)
+ (2, 4)
+
+ >>> version = "abc"
+ >>> majorMinorVersion(version) is None
+ True
+
+ """
+ m = re.search('^(\d+)\.(\d+)(\..*)?$', version)
+ if m is None:
+ return None
+ else:
+ return (int(m.group(1)), int(m.group(2)))
+
+
if __name__ == "__main__":
import doctest
(failure_count, test_count) = doctest.testmod()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org