Integration with Spark

["Zhenrui(Jerry) Zhang" <zh...@salesforce.com>]

Hi,

Does anyone has any updates on the integration with Spark. The feature
mentioned in https://drill.apache.org/blog/2014/12/16/whats-coming-in-2015/
and
http://www.slideshare.net/SparkSummit/adding-complex-data-to-spark-stackneeraja-rentachintala
? Also there is an issue opened in JIRA(
https://issues.apache.org/jira/browse/DRILL-3184) for this.

Thanks,
Jerry

Re: Integration with Spark

[Hanifi GUNES <ha...@gmail.com>]

I authored the initial Spark(DoS) plugin but it was never released due to
priorities. The initial implementation allows full duplex data exchange
between Drill and Spark. That is, we can use Drill to query your data lake
and do further iterative ML on it via Spark or vice versa.

I will need to rework in order to bring it to a publishable quality. Let me
know if you are interested in contributing.


-Hanifi

2016-05-27 11:23 GMT-07:00 Zhenrui(Jerry) Zhang <
zhenrui.zhang@salesforce.com>:

> Hi,
>
> Does anyone has any updates on the integration with Spark. The feature
> mentioned in https://drill.apache.org/blog/2014/12/16/whats-coming-in-
> 2015/
> and
> http://www.slideshare.net/SparkSummit/adding-complex-
> data-to-spark-stackneeraja-rentachintala
> ? Also there is an issue opened in JIRA(
> https://issues.apache.org/jira/browse/DRILL-3184) for this.
>
> Thanks,
> Jerry
>