You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "tigerhawkvok (via GitHub)" <gi...@apache.org> on 2024/10/25 16:54:04 UTC

[PR] [TYPING] Add type overloads for inplace dataframe operations [spark]

tigerhawkvok opened a new pull request, #48662:
URL: https://github.com/apache/spark/pull/48662

   Add type overloads for inplace dataframe operationsctly show a return of None or DataFrame based on flag state.
   
   Additionally adds one small type check function, `validate_strict_bool()` to wrap `validate_bool_kwargs()` for `inplace`, as `validate_bool_kwargs()` accepts `bool` or `None`; but `inplace` can only be `bool`.
   
   Aside from that very minor function edit, all other changes are exclusively to type hints.
   
   Simple test case:
   
   
   ```python
   from typing import Any, cast
   from pyspark.pandas import DataFrame
   
   foo = cast(DataFrame, None).sort_index()
   bar = cast(DataFrame, None).sort_index(inplace= True)
   
   bam = bool(cast(Any, None))
   baz = cast(DataFrame, None).sort_index(inplace= bam)
   ```
   
   
   Before:
   
   
   The union is returned regardless of state of `inplace`
   
   ![image](https://github.com/user-attachments/assets/9bb8f038-f6b8-4ce0-a078-851e78f35ac6)
   ![image](https://github.com/user-attachments/assets/6e78e59c-a1ad-4f5c-bfaa-f196ee418816)
   ![image](https://github.com/user-attachments/assets/fa24c56c-b11d-4152-9ad9-d179bd96ed19)
   
   
   
   
   After:
   
   - `inplace` is implicitly or explicitly `True` returns a `DataFrame`
   ![image](https://github.com/user-attachments/assets/cc8ee7b6-dacf-4dc5-a1d9-bed306fb6571)
   
   - `inplace` is explicitly `False` returns `None` 
   ![image](https://github.com/user-attachments/assets/0f515547-a8b0-48d3-8981-27de7acb6a7f)
   
   - `inplace` is ambiguous and either `True` or `False`, still returns `Union[DataFrame, None]`
   ![image](https://github.com/user-attachments/assets/6d6f7433-d66f-4eb1-ad44-298e0a9f33b0)
   ![image](https://github.com/user-attachments/assets/5bab3b56-2525-4f81-aee2-46556cefb8c6)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [TYPING] Add type overloads for inplace dataframe operations [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #48662:
URL: https://github.com/apache/spark/pull/48662#issuecomment-2451242444

   cc @itholic 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [TYPING] Add type overloads for inplace dataframe operations [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #48662: [TYPING] Add type overloads for inplace dataframe operations
URL: https://github.com/apache/spark/pull/48662


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [TYPING] Add type overloads for inplace dataframe operations [spark]

Posted by "tigerhawkvok (via GitHub)" <gi...@apache.org>.
tigerhawkvok commented on PR #48662:
URL: https://github.com/apache/spark/pull/48662#issuecomment-2439050357

   The lint error for this PR is an open MyPy bug:
   
   https://github.com/python/mypy/issues/11165


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [TYPING] Add type overloads for inplace dataframe operations [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48662:
URL: https://github.com/apache/spark/pull/48662#issuecomment-2646681728

   We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org