You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Gautam <ga...@gmail.com> on 2019/10/07 19:41:07 UTC
Iceberg Vectorized Reads Meeting Notes (Oct 7)
Hello Devs,
We met to discuss progress and next steps on Vectorized
read path in Iceberg. Here are my notes from the sync. Feel free to reply
with clarifications in case I mis-quoted or missed anything.
*Attendees*:
Anjali Norwood
Padma Pennumarthy
Ryan Blue
Samarth Jain
Gautam Kowshik
*Topics *
- Progress on Arrow Based Vectorization Reads
- Features being worked on and possible improvements
- Pending bottlenecks
- Identify things to collaborate on going forward.
- Next steps
Arrow Vectorized Reader
Samarth/Anjali:
- Working on Arrow based vectoization [1]
- At performance parity between Spark and Iceberg on primitive types
except strings.
- Planning to do dictionary encoding on strings
- New Arrow version gives boost in performance and fixes issues
- Vectorized batched Reading of definition levels improves performance
- Some checks had to be turned off in arrow to push performance further
viz. null check, unsafe memory access
- Implemented prefetching of parquet pages, this improves perf on
primitives beyond Vanilla spark
Ryan:
- Arrow version should not tied to spark and have iceberg specific
implementation binding so it will work with any reader not just spark.
- Add DatasourceV2Strategy to handle nested pruning into Spark upstream.
Will coordinate with Apple folks to add their work into Spark.
- Need ability to fallback to row based reads for cases where columnar
isn't possible. A config option maybe.
- Can add options where columnar batches are read into InternalRow and
returned to the Datasource.
Padma:
- Possibly contribute work on arrow back to arrow project. (can punt on
this for now to move forward faster on current work)
- Was looking into complex type support for Arrow based reads.
V1 Vectorized Read Path [2]
Gautam:
- Been working on V1 vectorized short circuit read path [3]. (this is
prolly not as useful once we have full featured support on Arrow based
reads)
- Will work on getting schema evolution parts working with this reader
by getting Projection unit/integration tests working. (this can be
contributed back into iceberg repo to unblock this path if we want to have
that option till arrow based read is fully capable)
*Next steps:*
- Unit tests for current Arrow based work.
- Provide options to perform vectorized batch reads, Row oriented reads
and Internal Row over Batch reads.
- Separate Arrow work in Iceberg into it's own sub-module
- Dictionary encoding support for strings in Arrow.
- Complex type support for Arrow.
- File issues for the above and identify how to distribute work between
us.
[1] https://github.com/apache/incubator-iceberg/tree/vectorized-read
[2] https://github.com/apache/incubator-iceberg/pull/462
[3]
https://github.com/prodeezy/incubator-iceberg/commits/v1-vectorized-reader
Open issues against Vectorized Iceberg Read milestone
Posted by Anjali Norwood <an...@netflix.com.INVALID>.
Thank you Gautam for the summary of the discussion.
Hello Devs,
The follow up vectorized iceberg tasks are now captured as issues against
the milestone.
Listed below for convenience.
https://github.com/apache/incubator-iceberg/issues/518
https://github.com/apache/incubator-iceberg/issues/519
https://github.com/apache/incubator-iceberg/issues/520
https://github.com/apache/incubator-iceberg/issues/521
https://github.com/apache/incubator-iceberg/issues/522
thanks,
Anjali.
On Mon, Oct 7, 2019 at 12:41 PM Gautam <ga...@gmail.com> wrote:
> Hello Devs,
> We met to discuss progress and next steps on Vectorized
> read path in Iceberg. Here are my notes from the sync. Feel free to reply
> with clarifications in case I mis-quoted or missed anything.
>
> *Attendees*:
>
> Anjali Norwood
> Padma Pennumarthy
> Ryan Blue
> Samarth Jain
> Gautam Kowshik
>
> *Topics *
> - Progress on Arrow Based Vectorization Reads
> - Features being worked on and possible improvements
> - Pending bottlenecks
> - Identify things to collaborate on going forward.
> - Next steps
>
> Arrow Vectorized Reader
>
> Samarth/Anjali:
>
> - Working on Arrow based vectoization [1]
> - At performance parity between Spark and Iceberg on primitive types
> except strings.
> - Planning to do dictionary encoding on strings
> - New Arrow version gives boost in performance and fixes issues
> - Vectorized batched Reading of definition levels improves performance
> - Some checks had to be turned off in arrow to push performance
> further viz. null check, unsafe memory access
> - Implemented prefetching of parquet pages, this improves perf on
> primitives beyond Vanilla spark
>
>
> Ryan:
>
>
> - Arrow version should not tied to spark and have iceberg specific
> implementation binding so it will work with any reader not just spark.
> - Add DatasourceV2Strategy to handle nested pruning into Spark
> upstream. Will coordinate with Apple folks to add their work into Spark.
> - Need ability to fallback to row based reads for cases where
> columnar isn't possible. A config option maybe.
> - Can add options where columnar batches are read into InternalRow and
> returned to the Datasource.
>
> Padma:
>
> - Possibly contribute work on arrow back to arrow project. (can punt
> on this for now to move forward faster on current work)
> - Was looking into complex type support for Arrow based reads.
>
>
> V1 Vectorized Read Path [2]
>
> Gautam:
>
> - Been working on V1 vectorized short circuit read path [3]. (this is
> prolly not as useful once we have full featured support on Arrow based
> reads)
> - Will work on getting schema evolution parts working with this reader
> by getting Projection unit/integration tests working. (this can be
> contributed back into iceberg repo to unblock this path if we want to have
> that option till arrow based read is fully capable)
>
>
>
> *Next steps:*
>
> - Unit tests for current Arrow based work.
> - Provide options to perform vectorized batch reads, Row oriented
> reads and Internal Row over Batch reads.
> - Separate Arrow work in Iceberg into it's own sub-module
> - Dictionary encoding support for strings in Arrow.
> - Complex type support for Arrow.
> - File issues for the above and identify how to distribute work
> between us.
>
>
>
>
> [1] https://github.com/apache/incubator-iceberg/tree/vectorized-read
>
> [2] https://github.com/apache/incubator-iceberg/pull/462
>
> [3]
> https://github.com/prodeezy/incubator-iceberg/commits/v1-vectorized-reader
>
>
>