[jira] [Commented] (DATAFU-180) Expose missing methods in Python

["Alon Hartanu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DATAFU-180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17929874#comment-17929874 ] 

Alon Hartanu commented on DATAFU-180:
-------------------------------------

Added code in [https://github.com/AlonH80/datafu] , not merged yet
python function dedup_by_all_except doesn't accept aggregate function at the moment - checking for possible solutions to pass aggregate function via python bridge

> Expose missing methods in Python
> --------------------------------
>
>                 Key: DATAFU-180
>                 URL: https://issues.apache.org/jira/browse/DATAFU-180
>             Project: DataFu
>          Issue Type: Improvement
>    Affects Versions: 1.7.0, 2.0.0, 1.8.0
>            Reporter: Eyal Allweil
>            Priority: Minor
>              Labels: good-first-issue, newbie, python, scala, up-for-grabs
>             Fix For: 2.1.0
>
>
> The [dedupRandomN|https://github.com/apache/datafu/blob/main/datafu-spark/src/main/scala/datafu/spark/SparkDFUtils.scala#L610] and [dedupByAllExcept|https://github.com/apache/datafu/blob/main/datafu-spark/src/main/scala/datafu/spark/SparkDFUtils.scala#L258] methods are not fully exposed in Python.
>  # They need to be added to [df_utils.py|https://github.com/apache/datafu/blob/6fd6fc4cb5e8156291600ee5f6ef3591dd74541e/datafu-spark/src/main/resources/pyspark_utils/df_utils.py]
>  # _dedupByAllExcept_ needs to be added to [SparkDFUtilsBridge.|https://github.com/apache/datafu/blob/main/datafu-spark/src/main/scala/datafu/spark/SparkDFUtils.scala#L35]
>  # The tests in [df_utils_tests.py|https://github.com/apache/datafu/blob/master/datafu-spark/src/test/resources/python_tests/df_utils_tests.py] should include both.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)