You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2018/03/23 04:20:45 UTC
spark git commit: [MINOR][PYTHON] Remove unused codes in schema
parsing logics of PySpark
Repository: spark
Updated Branches:
refs/heads/master 4d37008c7 -> a649fcf32
[MINOR][PYTHON] Remove unused codes in schema parsing logics of PySpark
## What changes were proposed in this pull request?
This PR proposes to remove out unused codes, `_ignore_brackets_split` and `_BRACKETS`.
`_ignore_brackets_split` was introduced in https://github.com/apache/spark/commit/d57daf1f7732a7ac54a91fe112deeda0a254f9ef to refactor and support `toDF("...")`; however, https://github.com/apache/spark/commit/ebc124d4c44d4c84f7868f390f778c0ff5cd66cb replaced the logics here. Seems `_ignore_brackets_split` is not referred anymore.
`_BRACKETS` was introduced in https://github.com/apache/spark/commit/880eabec37c69ce4e9594d7babfac291b0f93f50; however, all other usages were removed out in https://github.com/apache/spark/commit/648a8626b82d27d84db3e48bccfd73d020828586.
This is rather a followup for https://github.com/apache/spark/commit/ebc124d4c44d4c84f7868f390f778c0ff5cd66cb which I missed in that PR.
## How was this patch tested?
Manually tested. Existing tests should cover this. I also double checked by `grep` in the whole repo.
Author: hyukjinkwon <gu...@apache.org>
Closes #20878 from HyukjinKwon/minor-remove-unused.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a649fcf3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a649fcf3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a649fcf3
Branch: refs/heads/master
Commit: a649fcf32a7e610da2a2b4e3d94f5d1372c825d6
Parents: 4d37008
Author: hyukjinkwon <gu...@apache.org>
Authored: Thu Mar 22 21:20:41 2018 -0700
Committer: Wenchen Fan <we...@databricks.com>
Committed: Thu Mar 22 21:20:41 2018 -0700
----------------------------------------------------------------------
python/pyspark/sql/types.py | 35 -----------------------------------
1 file changed, 35 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/a649fcf3/python/pyspark/sql/types.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 826aab9..5d5919e 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -752,41 +752,6 @@ _all_complex_types = dict((v.typeName(), v)
_FIXED_DECIMAL = re.compile("decimal\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)")
-_BRACKETS = {'(': ')', '[': ']', '{': '}'}
-
-
-def _ignore_brackets_split(s, separator):
- """
- Splits the given string by given separator, but ignore separators inside brackets pairs, e.g.
- given "a,b" and separator ",", it will return ["a", "b"], but given "a<b,c>, d", it will return
- ["a<b,c>", "d"].
- """
- parts = []
- buf = ""
- level = 0
- for c in s:
- if c in _BRACKETS.keys():
- level += 1
- buf += c
- elif c in _BRACKETS.values():
- if level == 0:
- raise ValueError("Brackets are not correctly paired: %s" % s)
- level -= 1
- buf += c
- elif c == separator and level > 0:
- buf += c
- elif c == separator:
- parts.append(buf)
- buf = ""
- else:
- buf += c
-
- if len(buf) == 0:
- raise ValueError("The %s cannot be the last char: %s" % (separator, s))
- parts.append(buf)
- return parts
-
-
def _parse_datatype_string(s):
"""
Parses the given data type string to a :class:`DataType`. The data type string format equals
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org