You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/08/10 12:50:23 UTC
[GitHub] [iceberg] TGooch44 opened a new pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
TGooch44 opened a new pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314
Adding evaluator for determining if a given pyarrow parquet row group may contain rows for the assigned predicate and read-range[start, end]
cc: @danielcweeks @rdblue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rymurr commented on pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#issuecomment-671918121
> @rymurr, you might be interested in reviewing this one.
Very interested! Please give me a day or two as a bit busy at the moment.
I was looking at some similar stuff with the [Arrow Datasets](https://arrow.apache.org/docs/python/dataset.html) package.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#issuecomment-671663189
@rymurr, you might be interested in reviewing this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rymurr commented on a change in pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#discussion_r475222220
##########
File path: python/setup.py
##########
@@ -38,7 +37,7 @@
'requests',
'retrying',
'pandas',
- 'pyarrow'
+ 'pyarrow>=0.17.0'
Review comment:
any reason to not use 1.0?
##########
File path: python/iceberg/parquet/parquet_rowgroup_evaluator.py
##########
@@ -0,0 +1,340 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# inspiration drawn from the predicate accepts function in Kartothek
+# https://github.com/JDASoftwareGroup/kartothek/blob/master/kartothek/serialization/_parquet.py#L406
+
+from decimal import Decimal
+
+from iceberg.api.expressions import Binder, Expressions, ExpressionVisitors
+from iceberg.api.types import TypeID
+
+MICROSECOND_CONVERSION = 1000000
+
+
+class ParquetRowgroupEvaluator(object):
+ """
+ Evaluator for determining if a pyarrow parquet row-group matches the given expression and index bounds.
+
+ Parameters
+ ----------
+ schema : iceberg.api.Schema
+ An iceberg schema to use for binding the predicate
+ field_name_map: map
+ A map that translates file column names to the current schema
+ unbound : iceberg.api.expressions.UnboundPredicate
+ The unbound predicate to evaluate
+ start : int
+ The start index of the assigned reader
+ end : int
+ The end index of the assigned reader
+ """
+ def __init__(self, schema, field_name_map, unbound, start, end):
+ self.schema = schema
+ self.struct = schema.as_struct()
+ self.field_name_map = field_name_map
+
+ self.expr = None if unbound is None else Binder.bind(self.struct, Expressions.rewrite_not(unbound))
+ self.start = start
+ self.end = end
+ self._visitors = None
+
+ def _visitor(self):
+ if self._visitors is None:
+ self._visitors = ParquetRowgroupEvalVisitor(self.expr, self.schema, self.field_name_map,
+ self.struct, self.start, self.end)
+
+ return self._visitors
+
+ def eval(self, row_group):
+ return self._visitor().eval(row_group)
+
+
+class ParquetRowgroupEvalVisitor(ExpressionVisitors.BoundExpressionVisitor):
+ ROWS_MIGHT_MATCH = True
+ ROWS_CANNOT_MATCH = False
+
+ def __init__(self, expr, schema, field_name_map, struct, start, end):
+ self.expr = expr
+ self.schema = schema
+ self.field_name_map = field_name_map
+ self.struct = struct
+ self.start = start
+ self.end = end
+
+ # row-group stats info
+ self.lower_bounds = None
+ self.upper_bounds = None
+ self.nulls = None
+ self.num_rows = None
+ self.midpoint = None
+ self.parquet_cols = None
+
+ def eval(self, row_group):
+ """
+ Returns a boolean that determines if the given row-group may contain rows
+ for the assigned predicate and read-range[start, end]
+
+ Parameters
+ ----------
+ row_group : pyarrow._parquet.RowGroupMetaData
+ The pyarrow parquet row-group metadata being evaluated
+ Returns
+ -------
+ boolean
+ True if rows for the current evaluator might exist in the row group, false otherwise
+ """
+ if row_group.num_rows <= 0:
+ return ParquetRowgroupEvalVisitor.ROWS_CANNOT_MATCH
+
+ self.get_stats(row_group)
+ self.num_rows = row_group.num_rows
+
+ # if the mid-point of the row-group is not contained by the
+ # start-end range we don't read it
+ if self.start is not None and self.end is not None \
+ and not (self.start <= self.midpoint <= self.end):
+ return ParquetRowgroupEvalVisitor.ROWS_CANNOT_MATCH
+
+ if self.expr is None:
+ return ParquetRowgroupEvalVisitor.ROWS_MIGHT_MATCH
+
+ return ExpressionVisitors.visit(self.expr, self)
+
+ def always_true(self):
+ return ParquetRowgroupEvalVisitor.ROWS_MIGHT_MATCH
+
+ def always_false(self):
+ return ParquetRowgroupEvalVisitor.ROWS_CANNOT_MATCH
+
+ def not_(self, result):
+ return not result
+
+ def and_(self, left_result, right_result):
+ return left_result and right_result
+
+ def or_(self, left_result, right_result):
+ return left_result or right_result
+
+ def is_null(self, ref):
+ id = ref.field_id
Review comment:
nit: `id` variable hides the `id` builtin. Could we call this `field_id`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rymurr commented on pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#issuecomment-678975559
I doubt the Arrow Dataset code will have thought as long and as hard about schema evolution as you guys have! If you find any gaps in the dataset code I would love to help get the changes merged into Arrow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] TGooch44 closed pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
TGooch44 closed pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#issuecomment-679262477
I like using Arrow as much as possible. Like Ted said, the main concern is getting schema evolution right. Arrow/Parquet would need to implement ID-based schema evolution and at least some of the promotion rules like int -> long.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rymurr commented on pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#issuecomment-682418956
Hey @TGooch44. Twins! :exploding_head: Congrats and get some sleep! I am free from 11:30 GMT+1 if you are awake, else next week suits me perfectly fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rymurr commented on pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#issuecomment-678776115
apologies for the delay @TGooch44, looking now!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rymurr commented on pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#issuecomment-678798042
Regarding Arrows dataset here is a **very** rough example of what I think we can do.
The filters from this PR, partition filtering, different filesystem types etc are handled here. Partitions by iceberg metadata and the rest by pyarrow. What do you guys think?
``` python
from iceberg.hive import HiveTables
import pyarrow as pa
import pyarrow.dataset as ds
from pyarrow import fs
if __name__ == '__main__':
conf = {"hive.metastore.uris": 'thrift://localhost:9083',
'hive.metastore.warehouse.dir': '/home/ryan/warehouse/iceberg/hive_test'}
tables = HiveTables(conf)
tbl = tables.load("testing.foo")
# inspect metadata
print(tbl.schema())
print(tbl.spec())
print(int(tbl.current_snapshot().summary.get("total-records")))
scan = tbl.new_scan() \
.filter("symbol==AUDCHF") \
.select(["Bid", "Ask", "Datetime"])
projection = scan.schema
for task in scan.plan_tasks():
dataset = ds.FileSystemDataset.from_paths([i.file._file_path.replace("file:","") for i in task.files],
schema=pa.schema([("Bid", pa.float64()), ("Ask", pa.float64()),
("DateTime", pa.timestamp("us", 'UTC'))]),
format=ds.ParquetFileFormat(),
filesystem=fs.LocalFileSystem())
pytbl = dataset.to_table(filter=ds.field("Bid") > 0.75)
df = pytbl.to_pandas()
print(df)
```
Output
```
table {
1: DateTime: optional timestamptz(None)
2: Bid: optional double(None)
3: Ask: optional double(None)
4: symbol: optional string(None)
}
[
1000: DateTime_day: day(1)
]
846035
Bid Ask DateTime
0 0.75935 0.76156 2018-01-01 21:58:33.821000+00:00
1 0.75940 0.76155 2018-01-01 21:58:34.821000+00:00
2 0.75943 0.76154 2018-01-01 21:58:35.733000+00:00
3 0.75945 0.76153 2018-01-01 21:58:36.734000+00:00
4 0.75947 0.76152 2018-01-01 21:58:37.733000+00:00
... ... ... ...
388406 0.76612 0.76669 2018-01-05 21:59:00.070000+00:00
388407 0.76611 0.76669 2018-01-05 21:59:00.336000+00:00
388408 0.76611 0.76670 2018-01-05 21:59:00.809000+00:00
388409 0.76524 0.76747 2018-01-05 21:59:01.011000+00:00
388410 0.76534 0.76757 2018-01-05 21:59:12.367000+00:00
[388411 rows x 3 columns]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rymurr commented on a change in pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#discussion_r475412166
##########
File path: python/setup.py
##########
@@ -38,7 +37,7 @@
'requests',
'retrying',
'pandas',
- 'pyarrow'
+ 'pyarrow>=0.17.0'
Review comment:
welcome back!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] TGooch44 commented on pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
TGooch44 commented on pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#issuecomment-682122800
Next week may be better. I have twins that are still not really sleeping,
so my early mornings are a little touch and go. I have night help
some nights, which makes planning morning meetings a lot easier.
Otherwise, depending on your flexibility schedule-wise there is a good
chance I'll be up sometime between 12am-6am PDT, and could take a meeting.
Hard to commit to a specific time though.
On Thu, Aug 27, 2020 at 11:29 AM Ryan Murray <no...@github.com>
wrote:
> @rymurr <https://github.com/rymurr> I'm admittedly new to python type
> hints, so I took a crack at it, but would appreciate if you have a second
> to look it over. Also, re: Arrow Datasets, would you be opening to jumping
> on a quick video chat? I'd like to talk through it a bit with you if
> possible.
>
> Hey @TGooch44 <https://github.com/TGooch44> I am just knocking off for
> the day. I could do tomorrow your morning?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/iceberg/pull/1314#issuecomment-682117707>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAMETH4PL47ZJYV3TITRJA3SC2QYXANCNFSM4PZ5SJEQ>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] TGooch44 commented on pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
TGooch44 commented on pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#issuecomment-678804534
Let me review the additions to Dataset since it wasn't available when the bulk of our internal parquet code was written, the only reason for not more directly leveraging the arrow stuff was for enforcing schema evolution, eg. projecting nulls for columns that were added, renaming/reordering columns, etc.
I agree that this would work a lot better if the heavy lifting is done in the arrow C libs. I think with some tweaking we can probably make it happen.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rymurr commented on pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#issuecomment-682117707
> @rymurr I'm admittedly new to python type hints, so I took a crack at it, but would appreciate if you have a second to look it over. Also, re: Arrow Datasets, would you be opening to jumping on a quick video chat? I'd like to talk through it a bit with you if possible.
Hey @TGooch44 I am just knocking off for the day. I could do tomorrow your morning?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] TGooch44 commented on pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
TGooch44 commented on pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#issuecomment-673102463
I should have marked this is WIP. I'm still adding some logic to handle some of the other data types like timestamp. I'll try to get this updated soon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] TGooch44 commented on pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
TGooch44 commented on pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#issuecomment-682113308
@rymurr I'm admittedly new to python type hints, so I took a crack at it, but would appreciate if you have a second to look it over. Also, re: Arrow Datasets, would you be opening to jumping on a quick video chat? I'd like to talk through it a bit with you if possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] TGooch44 commented on pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
TGooch44 commented on pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#issuecomment-677651284
@rymurr This is ready for review whenever you have a chance. The failed check looks like it's something on the java side, which this PR isn't touching.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] TGooch44 commented on a change in pull request #1314: [python] Adding rowgroup evaluator using pyarrow parquet stats
Posted by GitBox <gi...@apache.org>.
TGooch44 commented on a change in pull request #1314:
URL: https://github.com/apache/iceberg/pull/1314#discussion_r475246257
##########
File path: python/setup.py
##########
@@ -38,7 +37,7 @@
'requests',
'retrying',
'pandas',
- 'pyarrow'
+ 'pyarrow>=0.17.0'
Review comment:
0.17.0 was the first release that contained field_id metadata necessary for schema evolution mapping. I'm just coming back from being on leave, so I'm a little behind on what's changed from 0.17.0->1.0, I'll try to get caught up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org