[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #8672: ARROW-10574: [Python][Parquet] Enhance hive partition filtering.

[GitBox <gi...@apache.org>]

jorisvandenbossche commented on a change in pull request #8672:
URL: https://github.com/apache/arrow/pull/8672#discussion_r527019353



##########
File path: python/pyarrow/parquet.py
##########
@@ -114,7 +115,24 @@ def _check_filters(filters, check_null_strings=True):
 
     Predicates may also be passed as List[Tuple]. This form is interpreted
     as a single conjunction. To express OR in predicates, one must
-    use the (preferred) List[List[Tuple]] notation."""
+    use the (preferred) List[List[Tuple]] notation.
+
+    Each tuple has format: (``key``, ``op``, ``value``) and compares the
+    ``key`` with the ``value``.
+    The supported ``op`` are:  ``=`` or ``==``, ``!=``, ``<``, ``>``, ``<=``,
+    ``>=``, ``in`` and ``not in``. If the ``op`` is ``in`` or ``not in``, the
+    ``value`` must be an iterable such as a ``list``, a ``set`` or a ``tuple``.
+
+    Examples:
+
+    .. code-block:: python
+
+        ('x', '=', 0)
+        ('y', 'in', ['a', 'b', 'c'])
+        ('z', 'not in', {'a','b'})
+
+

Review comment:
       A small nit: you can remove a blank line here

##########
File path: python/pyarrow/parquet.py
##########
@@ -871,21 +889,34 @@ def filter_accepts_partition(self, part_key, filter, level):
         if p_column != f_column:
             return True
 
-        f_type = type(f_value)
+        p_value = self.levels[level].dictionary[p_value_index].as_py()
 
-        if isinstance(f_value, set):
+        if op in {'in', 'not in'}:
+            if not (isinstance(f_value, Container) and
+                    isinstance(f_value, Iterable)):

Review comment:
       Can you use the abstract types from `collections.abc` ?  
   Also, we need to ensure we don't accidentally allow a single string value this way (`("part", "in", "abc")` would then be interpreted like `("part", "in", ["a", "b", "c"])`, which I don't think we should do)

##########
File path: python/pyarrow/parquet.py
##########
@@ -871,21 +889,34 @@ def filter_accepts_partition(self, part_key, filter, level):
         if p_column != f_column:
             return True
 
-        f_type = type(f_value)
+        p_value = self.levels[level].dictionary[p_value_index].as_py()
 
-        if isinstance(f_value, set):
+        if op in {'in', 'not in'}:
+            if not (isinstance(f_value, Container) and
+                    isinstance(f_value, Iterable)):
+                raise ValueError("Op '%s' requires an iterable such as list,"
+                                 " set or tuple", op)
             if not f_value:
-                raise ValueError("Cannot use empty set as filter value")
-            if op not in {'in', 'not in'}:
-                raise ValueError("Op '%s' not supported with set value",
-                                 op)
+                raise ValueError("Cannot use empty iterable as filter value")
+
             if len({type(item) for item in f_value}) != 1:
-                raise ValueError("All elements of set '%s' must be of"
+                raise ValueError("All elements of the iterable '%s' must be of"
                                  " same type", f_value)
             f_type = type(next(iter(f_value)))
 
-        p_value = f_type(self.levels[level]
-                         .dictionary[p_value_index].as_py())
+            p_value = f_type(p_value)
+
+            if op == 'in':
+                return p_value in f_value
+            else:
+                return p_value not in f_value

Review comment:
       Could you only do the `f_value` checking and `p_value` preparation in this `if` branch, but keep the actual check below?

##########
File path: python/pyarrow/parquet.py
##########
@@ -1237,6 +1264,8 @@ def validate_schemas(self):
             if self.common_metadata is not None:
                 self.schema = self.common_metadata.schema
             else:
+                if not self.pieces:
+                    raise ValueError('No partition is found.')

Review comment:
       Is this a related change? Why was it needed?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org