[jira] [Updated] (DATAFU-168) Add support for Spark 2.4.6 and up

["Eyal Allweil (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/DATAFU-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eyal Allweil updated DATAFU-168:
--------------------------------
    Fix Version/s: 1.8.0
                       (was: 1.7.0)

> Add support for Spark 2.4.6 and up
> ----------------------------------
>
>                 Key: DATAFU-168
>                 URL: https://issues.apache.org/jira/browse/DATAFU-168
>             Project: DataFu
>          Issue Type: Improvement
>    Affects Versions: 1.6.1
>            Reporter: Eyal Allweil
>            Priority: Major
>             Fix For: 1.8.0
>
>
> Once DATAFU-167 is merged, datafu-spark will support Spark versions up to 2.4.5. However, because our implementation of _collectLimitedList_ extends Spark's {_}collect{_}, and because its interface was changed in 2.4.6, compilation is broken for us.
>  
> (here is the relevant line from collectLimitedList: [https://github.com/apache/datafu/blob/master/datafu-spark/src/main/scala/spark/utils/overwrites/SparkOverwriteUDAFs.scala#L104)]
>  
> We need to either *1)* update our implementation, and drop support for older versions (and then release this in our version 1.8.0) or *2)* copy the code in a backwards compatible way.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)