You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jorisvandenbossche (via GitHub)" <gi...@apache.org> on 2023/07/07 14:15:59 UTC
[GitHub] [arrow] jorisvandenbossche opened a new pull request, #36550: GH-34884: [Python]: Support pickling pyarrow.dataset PartitioningFactory objects
jorisvandenbossche opened a new pull request, #36550:
URL: https://github.com/apache/arrow/pull/36550
### Rationale for this change
https://github.com/apache/arrow/pull/36462 already added support for pickling Partitioning objects, but not yet the PartitioningFactory objects.
The problem for PartitioningFactory is that we currently don't really expose the full class hierarchy in python, just the base class PartitioningFactory. We also don't expose creating those factory objects, except through the `discover` methods of the Partitioning classes.
I think it would be nice to keep this minimal binding, but that means if we want to make them serializable with pickle, we need another way to do that (and if we don't want to add custom code for serialization on the C++ side).
In this PR, I went for the route of essentially storing the constructor (the discover static method) and the arguments that were passed to the constructor, on the factory object, so we can use this info for pickling. Not the nicest code, but the simplest solution I could think of.
### Are these changes tested?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #36550: GH-34884: [Python]: Support pickling pyarrow.dataset PartitioningFactory objects
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on PR #36550:
URL: https://github.com/apache/arrow/pull/36550#issuecomment-1625631472
I just noticed that there is a minor lint failure in the CI. Then I think it's good to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche merged pull request #36550: GH-34884: [Python]: Support pickling pyarrow.dataset PartitioningFactory objects
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche merged PR #36550:
URL: https://github.com/apache/arrow/pull/36550
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #36550: GH-34884: [Python]: Support pickling pyarrow.dataset PartitioningFactory objects
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #36550:
URL: https://github.com/apache/arrow/pull/36550#discussion_r1256042524
##########
python/pyarrow/_dataset.pyx:
##########
@@ -2571,7 +2581,15 @@ cdef class DirectoryPartitioning(KeyValuePartitioning):
c_options.segment_encoding = _get_segment_encoding(segment_encoding)
return PartitioningFactory.wrap(
- CDirectoryPartitioning.MakeFactory(c_field_names, c_options))
+ CDirectoryPartitioning.MakeFactory(c_field_names, c_options),
+ _constructor_directory_partitioning_factory,
+ (field_names, infer_dictionary, max_partition_dictionary_size,
+ schema, segment_encoding)
+ )
+
+
+def _constructor_hive_partitioning_factory(*args):
+ return HivePartitioning.discover(*args)
Review Comment:
It just creates the factory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #36550: GH-34884: [Python]: Support pickling pyarrow.dataset PartitioningFactory objects
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #36550:
URL: https://github.com/apache/arrow/pull/36550#issuecomment-1625615402
> Seems like the biggest downside is that these _constructor_filename_partitioning_factory become visible symbols that users should ignore?
I prefixed them with an underscore, so by default those shouldn't be visible (eg with tab completion)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] conbench-apache-arrow[bot] commented on pull request #36550: GH-34884: [Python]: Support pickling pyarrow.dataset PartitioningFactory objects
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #36550:
URL: https://github.com/apache/arrow/pull/36550#issuecomment-1646194223
After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit a63ead769ae5c7e4f2a32f528bcb39c5aae2ab49.
There were 3 benchmark results indicating a performance regression:
- Commit Run on `ursa-i9-9960x` at [2023-07-16 19:09:53Z](http://conbench.ursa.dev/compare/runs/20dfbc9d815f4b48925b37c7857097c2...da62514664084835a8e6fb5904395c37/)
- [engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-15, scale_factor=1](http://conbench.ursa.dev/compare/benchmarks/064b431901487f4e8000ca70b2eb281f...064b45e9061473438000fdbadd99c833)
- [engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-16, scale_factor=1](http://conbench.ursa.dev/compare/benchmarks/064b431cd737719680009499cf82025c...064b45ee29c775d4800055514ae5d1a1)
- and 1 more (see the report linked below)
The [full Conbench report](https://github.com/apache/arrow/runs/15246710424) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #36550: GH-34884: [Python]: Support pickling pyarrow.dataset PartitioningFactory objects
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #36550:
URL: https://github.com/apache/arrow/pull/36550#discussion_r1256059272
##########
python/pyarrow/_dataset.pyx:
##########
@@ -2988,7 +3017,7 @@ cdef class FileSystemFactoryOptions(_Weakrefable):
c_factory = self.options.partitioning.factory()
if c_factory.get() == nullptr:
return None
- return PartitioningFactory.wrap(c_factory)
+ return PartitioningFactory.wrap(c_factory, None, None)
Review Comment:
Ok, I agree this is probably not needed. I don't think many users create `FileSystemFactoryOptions` (I don't think I even knew it existed :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #36550: GH-34884: [Python]: Support pickling pyarrow.dataset PartitioningFactory objects
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #36550:
URL: https://github.com/apache/arrow/pull/36550#discussion_r1256052302
##########
python/pyarrow/_dataset.pyx:
##########
@@ -2988,7 +3017,7 @@ cdef class FileSystemFactoryOptions(_Weakrefable):
c_factory = self.options.partitioning.factory()
if c_factory.get() == nullptr:
return None
- return PartitioningFactory.wrap(c_factory)
+ return PartitioningFactory.wrap(c_factory, None, None)
Review Comment:
Yes, that's correct.
But this is only if you first create an options object with a factory, and then decide the pickle the factory accessed from the options object instead of the factory you created to start with:
```python
partitioning_factory = ds.HivePartitioning.discover(..)
options = ds.FileSystemFactoryOptions(partitioning=partitioning_factory)
pickle.dumps(options.partitioning_factory) # instead of pickle.dumps(partitioning_factory)
```
So in practice, if you run into this, you can always pickle the original factory object you created (as there is no other way to get such an object than creating it yourself, i.e. the options object only has it when you passed it manually).
We _could_ support pickling those options-accessed factory objects (and ideally that would also be the case, for consistency / avoiding confusion), but the current code only stores the C++ factory object on the options class and recreates the python factory object from the C++ object, and then we don't know anymore how this C++ object was created. We could add some workaround to also store the python factory object, but I am not sure this is worth the effort, given that as a user you can always pickle the original object (explanation above)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #36550: GH-34884: [Python]: Support pickling pyarrow.dataset PartitioningFactory objects
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #36550:
URL: https://github.com/apache/arrow/pull/36550#discussion_r1256029513
##########
python/pyarrow/_dataset.pyx:
##########
@@ -2988,7 +3017,7 @@ cdef class FileSystemFactoryOptions(_Weakrefable):
c_factory = self.options.partitioning.factory()
if c_factory.get() == nullptr:
return None
- return PartitioningFactory.wrap(c_factory)
+ return PartitioningFactory.wrap(c_factory, None, None)
Review Comment:
Won't this mean that this object can't be pickled?
##########
python/pyarrow/_dataset.pyx:
##########
@@ -2571,7 +2581,15 @@ cdef class DirectoryPartitioning(KeyValuePartitioning):
c_options.segment_encoding = _get_segment_encoding(segment_encoding)
return PartitioningFactory.wrap(
- CDirectoryPartitioning.MakeFactory(c_field_names, c_options))
+ CDirectoryPartitioning.MakeFactory(c_field_names, c_options),
+ _constructor_directory_partitioning_factory,
+ (field_names, infer_dictionary, max_partition_dictionary_size,
+ schema, segment_encoding)
+ )
+
+
+def _constructor_hive_partitioning_factory(*args):
+ return HivePartitioning.discover(*args)
Review Comment:
Does this method actually run discovery or does it just create the factory?
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -805,7 +805,7 @@ cdef class ParquetFactoryOptions(_Weakrefable):
c_factory = self.options.partitioning.factory()
if c_factory.get() == nullptr:
return None
- return PartitioningFactory.wrap(c_factory)
+ return PartitioningFactory.wrap(c_factory, None, None)
Review Comment:
Same question as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org