You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Julian-J-S (via GitHub)" <gi...@apache.org> on 2026/05/15 14:59:52 UTC

[I] Pyspark: `from_json` is missing `MapType` as a type hint for `schema` argument [spark]

Julian-J-S opened a new issue, #55900:
URL: https://github.com/apache/spark/issues/55900

   Using pyspark we always try to use concrete types everywhere instead of strings to improve quality and maintainability.
   
   We recently came across this:
   ```python
   from pyspark.sql import functions as F
   from pyspark.sql import types as T
   
   _ = F.from_json(
       F.col("value"),
       schema=T.MapType(T.StringType(), T.StringType())  # >>> TYPE ERROR
   )
   ```
   This code works! ✅ 
   
   However, type checkers are not happy: 🛑 
   Type Error: Argument to function `from_json` is incorrect: Expected `ArrayType | StructType | Column | str`, found `MapType`.
   
   
   Is there a reason that `MapType` is not included as a type hint?
   There is even a concrete example using `MAP` in the function documentation of `from_json`:
   > df.select(sf.from_json(df.value, "MAP<STRING,INT>").alias("json")).show()


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] Pyspark: `from_json` is missing `MapType` as a type hint for `schema` argument [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #55900: Pyspark: `from_json` is missing `MapType` as a type hint for `schema` argument
URL: https://github.com/apache/spark/issues/55900


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] Pyspark: `from_json` is missing `MapType` as a type hint for `schema` argument [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on issue #55900:
URL: https://github.com/apache/spark/issues/55900#issuecomment-4483600985

   > [@zhengruifeng](https://github.com/zhengruifeng) do you know a specific reason that we do or do not support `MapType` (or other `DataType`) for `from_json`?
   
   @gaogaotiantian I guess the MapType was just missed, when the SQL side accepted MapType the python doc was not updated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] Pyspark: `from_json` is missing `MapType` as a type hint for `schema` argument [spark]

Posted by "deepanshu36 (via GitHub)" <gi...@apache.org>.
deepanshu36 commented on issue #55900:
URL: https://github.com/apache/spark/issues/55900#issuecomment-4469996795

   hi i have contributed to this change , please have a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] Pyspark: `from_json` is missing `MapType` as a type hint for `schema` argument [spark]

Posted by "puneetdixit200 (via GitHub)" <gi...@apache.org>.
puneetdixit200 commented on issue #55900:
URL: https://github.com/apache/spark/issues/55900#issuecomment-4468449587

   Hello maintainers, I would like to work on this PySpark typing issue. I can update the from_json type annotations, add typing coverage for MapType schemas, and verify compatibility with existing string and StructType schema usage. Please assign this to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] Pyspark: `from_json` is missing `MapType` as a type hint for `schema` argument [spark]

Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on issue #55900:
URL: https://github.com/apache/spark/issues/55900#issuecomment-4481581485

   @zhengruifeng do you know a specific reason that we do or do not support `MapType` (or other `DataType`) for `from_json`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] Pyspark: `from_json` is missing `MapType` as a type hint for `schema` argument [spark]

Posted by "deepanshu36 (via GitHub)" <gi...@apache.org>.
deepanshu36 commented on issue #55900:
URL: https://github.com/apache/spark/issues/55900#issuecomment-4483107634

   > [@zhengruifeng](https://github.com/zhengruifeng) do you know a specific reason that we do or do not support `MapType` (or other `DataType`) for `from_json`?
   
   i looked up over the internet on how this can be added to our code so that the type checkers do not throw the error. Not exactly sure why it wasn't present earlier.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org