[jira] [Commented] (DATAFU-148) Setup Spark sub-project

["Eyal Allweil (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DATAFU-148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628836#comment-16628836 ] 

Eyal Allweil commented on DATAFU-148:
-------------------------------------

Hi [~matterhayes] - thanks for your review!

Regarding the changes for gradle - I agree, and I'll implement them and push them into our fork.

Regarding flatten, changeSchema and filterOut - the question is, do we want them? If so, we'll obviously prepare tests and documentation - I just wasn't sure whether they should be included or not.

The reason for the separations in SparkDFUtils and DataFrameOps has to do with exposing these methods to Python - that's code we haven't gotten around to preparing yet.

I also agree that we can/should merge this into a 2.0.0 branch.

[~uzadude] - anything to add?

> Setup Spark sub-project
> -----------------------
>
>                 Key: DATAFU-148
>                 URL: https://issues.apache.org/jira/browse/DATAFU-148
>             Project: DataFu
>          Issue Type: New Feature
>            Reporter: Eyal Allweil
>            Assignee: Eyal Allweil
>            Priority: Major
>
> Create a skeleton Spark sub project for Spark code to be contributed to DataFu



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)