[I] GeoPandas: GeoSeries constructor fails for leading-null local input on Spark 3.5 [sedona]

["jiayuasu (via GitHub)" <gi...@apache.org>]

jiayuasu opened a new issue, #3333:
URL: https://github.com/apache/sedona/issues/3333

   ### Expected behavior
   
   `GeoSeries` should accept local geometry data with leading nulls, preserving the nulls, geometries, and row order. Construction should not depend on whether the first row contains a geometry.
   
   ### Actual behavior
   
   On Spark 3.5.4, both of these fail during construction, before any spatial operation runs:
   
   ```python
   GeoSeries([None, Point(1, 0)])
   GeoSeries(gpd.GeoSeries([None, Point(1, 0)]))
   ```
   
   The error is:
   
   ```text
   pyarrow.lib.ArrowInvalid: Could not convert <POINT (1 0)> with type Point: did not recognize Python value type when inferring an Arrow data type
   ```
   
   The equivalent non-null-first inputs and `[None, None]` succeed. All five cases also succeed with Spark 4.1.1 and the same Python dependencies. Other Spark versions were not tested in this reproduction.
   
   ### Steps to reproduce
   
   In a Sedona-enabled Spark 3.5.4 environment:
   
   ```python
   import geopandas as gpd
   from shapely.geometry import Point
   from sedona.spark import SedonaContext
   
   spark = SedonaContext.create(
       SedonaContext.builder()
       .master("local[2]")
       .config("spark.sql.ansi.enabled", "false")
       .getOrCreate()
   )
   
   from sedona.spark.geopandas import GeoSeries
   
   # Fails during construction.
   GeoSeries([None, Point(1, 0)])
   
   # Run separately: also fails.
   GeoSeries(gpd.GeoSeries([None, Point(1, 0)]))
   
   # Controls: both succeed.
   GeoSeries([Point(1, 0), None]).to_geopandas()
   GeoSeries([None, None]).to_geopandas()
   ```
   
   ### Cause and scope
   
   The local-input path in `GeoSeries.__init__` converts the input to an object-typed pandas Series and delegates to pandas-on-Spark. In Spark 3.5.4, `infer_pd_series_spark_type` checks only `pser.iloc[0]` for the geometry's `__UDT__` attribute. A leading null bypasses that check, so it tries Arrow inference on the Shapely values and fails.
   
   Spark 4.1.1 checks the first non-null value instead, which explains the passing control run.
   
   This was found while reviewing #3302, but is a separate constructor bug. It reproduces without calling `fillna`; the constructor is unchanged by that PR. A fix should preserve index, name, CRS, and input order, with regression coverage for leading-null local inputs on Spark 3.5 as well as the existing non-null-first and all-null cases.
   
   ### Sedona version
   
   Development Python source at `b3bd16f87eb32de9fda40ca43ab1e4c8a18e48d3`. The constructor is identical to master at `f000c4346cdcaadcee17e2ea74e2ef57949a3e52`. The Spark 3.5 reproduction used the Sedona 1.9.1 shaded JAR for Spark 3.5 / Scala 2.12.
   
   ### Apache Spark version
   
   Reproduced on PySpark 3.5.4; passing control on PySpark 4.1.1.
   
   ### API type
   
   Python
   
   ### JRE version
   
   17.0.13
   
   ### Python version
   
   3.11.13
   
   ### Environment
   
   macOS arm64, local Spark. pandas 2.2.3, GeoPandas 1.1.4, Shapely 2.1.2, PyArrow 16.1.0.
   
   ### Existing issues
   
   - [x] I searched the existing issues and did not find a duplicate.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] GeoPandas: GeoSeries constructor fails for leading-null local input on Spark 3.5 [sedona]

["jiayuasu (via GitHub)" <gi...@apache.org>]

jiayuasu closed issue #3333: GeoPandas: GeoSeries constructor fails for leading-null local input on Spark 3.5
URL: https://github.com/apache/sedona/issues/3333


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org