You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Henry Davidge (Jira)" <ji...@apache.org> on 2019/09/03 18:39:00 UTC

[jira] [Created] (SPARK-28962) High-order function: filter(array, function) → array

Henry Davidge created SPARK-28962:
-------------------------------------

             Summary: High-order function: filter(array<T>, function<T, Int, boolean>) → array<T>
                 Key: SPARK-28962
                 URL: https://issues.apache.org/jira/browse/SPARK-28962
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: Henry Davidge


It's helpful to have access to the index when using the {{filter}} function. For instance, we're using SparkSQL to manipulate genomic data. We store some fields in a long array that has the same length for every row in the DataFrame. We compute aggregates that are per array position (so we look at the kth element for each row's array) and then want to filter each row's array by looking values in the aggregate array.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org