[jira] [Commented] (SPARK-55507) is_geometry and is_geography crash with TypeError when struct field has no metadata

["Uroš Bojanić (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-55507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18058326#comment-18058326 ] 

Uroš Bojanić commented on SPARK-55507:
--------------------------------------

Hi [~yiconghuang], thank you for helping out the Geo effort! Please move this ticket into: https://issues.apache.org/jira/browse/SPARK-51658.

> is_geometry and is_geography crash with TypeError when struct field has no metadata
> -----------------------------------------------------------------------------------
>
>                 Key: SPARK-55507
>                 URL: https://issues.apache.org/jira/browse/SPARK-55507
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 4.2.0
>            Reporter: Yicong Huang
>            Priority: Minor
>              Labels: pull-request-available
>
> {{is_geometry()}} and {{is_geography()}} in {{python/pyspark/sql/pandas/types.py}} crash when a PyArrow struct contains a field named {{wkb}} without metadata.
> PyArrow fields have {{metadata=None}} by default (not an empty dict). The current code does:
> {code:python}
> field.name == "wkb"
> and b"geometry" in field.metadata  # TypeError if metadata is None
> and field.metadata[b"geometry"] == b"true"
> {code}
> This raises {{TypeError: argument of type 'NoneType' is not iterable}}.
> These functions are called from {{from_arrow_type()}} for every struct type, which is used in {{toPandas()}}, {{pandas_udf}}, and Spark Connect schema parsing. Any user with a struct containing a field named {{wkb}} without geometry metadata will hit this crash.
> The fix is to add a {{field.metadata is not None}} check before the {{in}} operator:
> {code:python}
> field.name == "wkb"
> and field.metadata is not None
> and b"geometry" in field.metadata
> and field.metadata[b"geometry"] == b"true"
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org