You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@datafu.apache.org by "Eyal Allweil (Jira)" <ji...@apache.org> on 2022/10/24 07:28:00 UTC

[jira] [Created] (DATAFU-168) Add support for Spark 2.4.6 and up

Eyal Allweil created DATAFU-168:
-----------------------------------

             Summary: Add support for Spark 2.4.6 and up
                 Key: DATAFU-168
                 URL: https://issues.apache.org/jira/browse/DATAFU-168
             Project: DataFu
          Issue Type: Improvement
    Affects Versions: 1.6.1
            Reporter: Eyal Allweil
             Fix For: 1.7.0


Once DATAFU-167 is merged, datafu-spark will support Spark versions up to 2.4.5. However, because our implementation of _collectLimitedList_ extends Spark's {_}collect{_}, and because its interface was changed in 2.4.6, compilation is broken for us.

 

(here is the relevant line from collectLimitedList: [https://github.com/apache/datafu/blob/master/datafu-spark/src/main/scala/spark/utils/overwrites/SparkOverwriteUDAFs.scala#L104)]

 

We need to either *1)* update our implementation, and drop support for older versions (and then release this in our version 1.8.0) or *2)* copy the code in a backwards compatible way.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)