You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Gautam <ga...@gmail.com> on 2019/10/07 19:41:07 UTC

Iceberg Vectorized Reads Meeting Notes (Oct 7)

Hello Devs,
                We met to discuss progress and next steps on Vectorized
read path in Iceberg. Here are my notes from the sync. Feel free to reply
with clarifications in case I mis-quoted or missed anything.

*Attendees*:

Anjali Norwood
Padma Pennumarthy
Ryan Blue
Samarth Jain
Gautam Kowshik

*Topics *
- Progress on Arrow Based Vectorization Reads
- Features being worked on and possible improvements
- Pending bottlenecks
- Identify things to collaborate on going forward.
- Next steps

Arrow Vectorized Reader

  Samarth/Anjali:

   - Working on Arrow based vectoization [1]
   - At  performance parity between Spark and Iceberg on primitive types
   except strings.
   - Planning to do dictionary encoding on strings
   - New Arrow version gives boost in performance and fixes issues
   - Vectorized batched Reading of definition levels improves performance
   - Some checks had to be turned off in arrow to push performance further
   viz. null check, unsafe memory access
   - Implemented prefetching of parquet pages, this improves perf on
   primitives beyond Vanilla spark


   Ryan:


   - Arrow version should not tied to spark and have iceberg specific
   implementation binding so it will work with any reader not just spark.
   - Add DatasourceV2Strategy to handle nested pruning into Spark upstream.
   Will coordinate with Apple folks to add their work into Spark.
   - Need ability to fallback  to row based reads for cases where columnar
   isn't possible. A config option maybe.
   - Can add options where columnar batches are read into InternalRow and
   returned to the Datasource.

  Padma:

   - Possibly contribute work on arrow back to arrow project. (can punt on
   this for now to move forward faster on current work)
   - Was looking into complex type support for Arrow based reads.


V1 Vectorized Read Path [2]

Gautam:

   - Been working on V1 vectorized short circuit read path [3]. (this is
   prolly not as useful once we have full featured support on Arrow based
   reads)
   - Will work on getting schema evolution parts working with this reader
   by getting Projection unit/integration tests working. (this can be
   contributed back into iceberg repo to unblock this path if we want to have
   that option till arrow based read is fully capable)



*Next steps:*

   - Unit tests for current Arrow based work.
   - Provide options to perform vectorized batch reads, Row oriented reads
   and Internal Row over Batch reads.
   - Separate Arrow work in Iceberg into it's own sub-module
   - Dictionary encoding support for strings in Arrow.
   - Complex type support for Arrow.
   - File issues for the above and identify how to distribute work between
   us.




[1]  https://github.com/apache/incubator-iceberg/tree/vectorized-read

[2]  https://github.com/apache/incubator-iceberg/pull/462

[3]
https://github.com/prodeezy/incubator-iceberg/commits/v1-vectorized-reader

Open issues against Vectorized Iceberg Read milestone

Posted by Anjali Norwood <an...@netflix.com.INVALID>.
Thank you Gautam for the summary of the discussion.

Hello Devs,

The follow up vectorized iceberg tasks are now captured as issues against
the milestone.
Listed below for convenience.
https://github.com/apache/incubator-iceberg/issues/518
https://github.com/apache/incubator-iceberg/issues/519
https://github.com/apache/incubator-iceberg/issues/520
https://github.com/apache/incubator-iceberg/issues/521
https://github.com/apache/incubator-iceberg/issues/522

thanks,
Anjali.


On Mon, Oct 7, 2019 at 12:41 PM Gautam <ga...@gmail.com> wrote:

> Hello Devs,
>                 We met to discuss progress and next steps on Vectorized
> read path in Iceberg. Here are my notes from the sync. Feel free to reply
> with clarifications in case I mis-quoted or missed anything.
>
> *Attendees*:
>
> Anjali Norwood
> Padma Pennumarthy
> Ryan Blue
> Samarth Jain
> Gautam Kowshik
>
> *Topics *
> - Progress on Arrow Based Vectorization Reads
> - Features being worked on and possible improvements
> - Pending bottlenecks
> - Identify things to collaborate on going forward.
> - Next steps
>
> Arrow Vectorized Reader
>
>   Samarth/Anjali:
>
>    - Working on Arrow based vectoization [1]
>    - At  performance parity between Spark and Iceberg on primitive types
>    except strings.
>    - Planning to do dictionary encoding on strings
>    - New Arrow version gives boost in performance and fixes issues
>    - Vectorized batched Reading of definition levels improves performance
>    - Some checks had to be turned off in arrow to push performance
>    further viz. null check, unsafe memory access
>    - Implemented prefetching of parquet pages, this improves perf on
>    primitives beyond Vanilla spark
>
>
>    Ryan:
>
>
>    - Arrow version should not tied to spark and have iceberg specific
>    implementation binding so it will work with any reader not just spark.
>    - Add DatasourceV2Strategy to handle nested pruning into Spark
>    upstream. Will coordinate with Apple folks to add their work into Spark.
>    - Need ability to fallback  to row based reads for cases where
>    columnar isn't possible. A config option maybe.
>    - Can add options where columnar batches are read into InternalRow and
>    returned to the Datasource.
>
>   Padma:
>
>    - Possibly contribute work on arrow back to arrow project. (can punt
>    on this for now to move forward faster on current work)
>    - Was looking into complex type support for Arrow based reads.
>
>
> V1 Vectorized Read Path [2]
>
> Gautam:
>
>    - Been working on V1 vectorized short circuit read path [3]. (this is
>    prolly not as useful once we have full featured support on Arrow based
>    reads)
>    - Will work on getting schema evolution parts working with this reader
>    by getting Projection unit/integration tests working. (this can be
>    contributed back into iceberg repo to unblock this path if we want to have
>    that option till arrow based read is fully capable)
>
>
>
> *Next steps:*
>
>    - Unit tests for current Arrow based work.
>    - Provide options to perform vectorized batch reads, Row oriented
>    reads and Internal Row over Batch reads.
>    - Separate Arrow work in Iceberg into it's own sub-module
>    - Dictionary encoding support for strings in Arrow.
>    - Complex type support for Arrow.
>    - File issues for the above and identify how to distribute work
>    between us.
>
>
>
>
> [1]  https://github.com/apache/incubator-iceberg/tree/vectorized-read
>
> [2]  https://github.com/apache/incubator-iceberg/pull/462
>
> [3]
> https://github.com/prodeezy/incubator-iceberg/commits/v1-vectorized-reader
>
>
>