You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Oscar Boykin (JIRA)" <ji...@apache.org> on 2015/03/16 18:35:39 UTC

[jira] [Commented] (PARQUET-218) Macro based Scala APIs for Predicate and Projection push down

    [ https://issues.apache.org/jira/browse/PARQUET-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363558#comment-14363558 ] 

Oscar Boykin commented on PARQUET-218:
--------------------------------------

I think these macros are great, but I'd like to see if we can go a bit further.

I'd like Projection[T, U] to extend (or be implicitly convertible to Function1[T, U]) so that we can use projections in scalding/spark without being sure the underlying data supports parquet project (we have sources that are mixed: for some time ranges they are parquet, but older data might just be compressed thrift).

If we have this functionality, we can project as the IO layer when possible, when not possible we can do it at scalding/spark layer.

> Macro based Scala APIs for Predicate and Projection push down
> -------------------------------------------------------------
>
>                 Key: PARQUET-218
>                 URL: https://issues.apache.org/jira/browse/PARQUET-218
>             Project: Parquet
>          Issue Type: Improvement
>          Components: parquet-mr
>            Reporter: Julien Le Dem
>
> There has been some experiments with Scala macros to make Predicate and Projection APIs in Scala.
> In particular by [~neville]:
> https://github.com/nevillelyh/parquet-avro-extra/blob/master/parquet-avro-examples/src/main/scala/me/lyh/parquet/avro/examples/ExampleApp.scala
> https://github.com/nevillelyh/parquet-avro-extra/commit/090f8d1d1d7d44489bb71e04465d4a12a46432c9#diff-89dca6adcdf105c6063661de43931c29R15
> There have also been some discussion at Twitter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)