You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/08/26 23:20:24 UTC
[GitHub] [arrow] westonpace opened a new pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
westonpace opened a new pull request #11014:
URL: https://github.com/apache/arrow/pull/11014
This PR ended up being fairly extensive but mostly to avoid a single `shared_ptr` copy (due to changing the constructor to take in `const Schema& schema` instead of `std::shared_ptr<Schema>` since I now copy the field names and types out of the schema) and I'm not entirely sure that's worth it over just keeping the old interface.
Otherwise the only potential downside is that `std::shared_ptr<Schema> Partitioning::schema()` became `Result<std::shared_ptr<Schema>> Partitioning::GetSchema()` since it could potentially fail if the partitioning was created without a schema. I also removed the corresponding accessor from python.
I don't think this is a problem though because `GetSchema` would only work if there was a schema which means...
1. You created the partitioning with a schema in the first place so there is no need to pull it back out.
2. You created the schema with a FileSystemDatasetFactory so you can just get the field types from the dataset schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11014:
URL: https://github.com/apache/arrow/pull/11014#issuecomment-906807010
https://issues.apache.org/jira/browse/ARROW-13775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace closed pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
westonpace closed pull request #11014:
URL: https://github.com/apache/arrow/pull/11014
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #11014:
URL: https://github.com/apache/arrow/pull/11014#issuecomment-908602103
@jorisvandenbossche Ok, I had to take a further look at the PR. I didn't quite realize what you are doing. You are allowing the write_dataset method to accept a partitioning factory which you then populate with the schema from the table being written. I'll withdraw this then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #11014:
URL: https://github.com/apache/arrow/pull/11014#issuecomment-908601345
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #11014:
URL: https://github.com/apache/arrow/pull/11014#issuecomment-908608144
To be clear, it's certainly a bit of "misuse" of the PartitioningFactory object, as that is meant to "finish" a partitioning based on parsing file names to infer the schema. But it's also a smaller change, so that might be simpler than changing the Partitioning interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #11014:
URL: https://github.com/apache/arrow/pull/11014#issuecomment-908595122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #11014:
URL: https://github.com/apache/arrow/pull/11014#issuecomment-908099090
Can you explain a bit what's the goal for allowing a Partitioning object without a schema? (we already have a PartitioningFactory for the case where the schema gets inferred from parsed paths)
What would be a use case? (I don't see any test where it is actually used for something) I suppose as alternative for https://github.com/apache/arrow/pull/11008, right?
> I also removed the corresponding accessor from python. I don't think this is a problem though because `GetSchema` would only work if there was a schema which means...
>
> 1. You created the partitioning with a schema in the first place so there is no need to pull it back out.
> 2. You created the schema with a FileSystemDatasetFactory so you can just get the field types from the dataset schema.
I added the partitioning attribute on a Dataset on purpose so you can easily check the schema of your partitioning after reading a partitioned dataset (eg dask needs this). It's true that you also can get the types of the partition columns from the dataset's schema if you still have access to the field names of the Partitioning (but this doesn't yet seem to be exposed in this PR, though it should be easy to add), but that's certainly less convenient IMO.
(We could also still leave the attribute, but have it error if not available)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace closed pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
westonpace closed pull request #11014:
URL: https://github.com/apache/arrow/pull/11014
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #11014:
URL: https://github.com/apache/arrow/pull/11014#issuecomment-908496393
> This PR ended up being fairly extensive but mostly to avoid a single `shared_ptr` copy (due to changing the constructor to take in `const Schema& schema` instead of `std::shared_ptr<Schema>` since I now copy the field names and types out of the schema) and I'm not entirely sure that's worth it over just keeping the old interface.
Just take `const std::shared_ptr<Schema>&` instead to minimize changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #11014:
URL: https://github.com/apache/arrow/pull/11014#issuecomment-908496393
> This PR ended up being fairly extensive but mostly to avoid a single `shared_ptr` copy (due to changing the constructor to take in `const Schema& schema` instead of `std::shared_ptr<Schema>` since I now copy the field names and types out of the schema) and I'm not entirely sure that's worth it over just keeping the old interface.
Just take `const std::shared_ptr<Schema>&` instead to minimize changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #11014:
URL: https://github.com/apache/arrow/pull/11014#issuecomment-908601345
> A partitioning factory cannot be used for writing datasets (only reading)...
Yes, that's the case right now, but that's what #11008 changes: under the hood in `ds.write_dataset(..)` it combines the PartitioningFactory (if passed by the user) with the schema of the dataset to get an actual Partitioning, which can then be used for writing the dataset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace edited a comment on pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
westonpace edited a comment on pull request #11014:
URL: https://github.com/apache/arrow/pull/11014#issuecomment-908595122
> What would be a use case? (I don't see any test where it is actually used for something) I suppose as alternative for #11008, right?
@jorisvandenbossche
Yes, but not as an alternative, this would be to enable #11008. I don't believe #11008 works today. `ds.partitioning(field_names=['a'])` returns a partitioning factory and not a partitioning. It also returns an error when specifying the hive format. A partitioning factory cannot be used for writing datasets (only reading)...
```
>>> import pyarrow
>>> import pyarrow as pa
>>> import pyarrow.dataset as ds
>>> table = pa.Table.from_pydict({'a': [1, 2, 3], 'b': ['x', 'y', 'z']})
>>> ds.partitioning(field_names=['a'])
<pyarrow._dataset.PartitioningFactory object at 0x7f2410d4c170>
>>> part = ds.partitioning(field_names=['a'])
>>> ds.write_dataset(table, '/tmp/new_dataset', format='parquet', partitioning=part)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pace/anaconda3/envs/arrow-release-5/lib/python3.9/site-packages/pyarrow/dataset.py", line 791, in write_dataset
partitioning = _ensure_write_partitioning(partitioning)
File "/home/pace/anaconda3/envs/arrow-release-5/lib/python3.9/site-packages/pyarrow/dataset.py", line 686, in _ensure_write_partitioning
raise ValueError("partitioning needs to be actual Partitioning object")
ValueError: partitioning needs to be actual Partitioning object
```
I didn't add real world use cases (or python bindings) because I figured those would be covered by #11008.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace edited a comment on pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
westonpace edited a comment on pull request #11014:
URL: https://github.com/apache/arrow/pull/11014#issuecomment-908595122
> What would be a use case? (I don't see any test where it is actually used for something) I suppose as alternative for #11008, right?
@jorisvandenbossche
Yes, but not as an alternative, this would be to enable #11008. I don't believe #11008 works today. `ds.partitioning(field_names=['a'])` returns a partitioning factory and not a partitioning. It also returns an error when specifying the hive format. A partitioning factory cannot be used for writing datasets (only reading)...
```
>>> import pyarrow
>>> import pyarrow as pa
>>> import pyarrow.dataset as ds
>>> table = pa.Table.from_pydict({'a': [1, 2, 3], 'b': ['x', 'y', 'z']})
>>> ds.partitioning(field_names=['a'])
<pyarrow._dataset.PartitioningFactory object at 0x7f2410d4c170>
>>> part = ds.partitioning(field_names=['a'])
>>> ds.write_dataset(table, '/tmp/new_dataset', format='parquet', partitioning=part)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pace/anaconda3/envs/arrow-release-5/lib/python3.9/site-packages/pyarrow/dataset.py", line 791, in write_dataset
partitioning = _ensure_write_partitioning(partitioning)
File "/home/pace/anaconda3/envs/arrow-release-5/lib/python3.9/site-packages/pyarrow/dataset.py", line 686, in _ensure_write_partitioning
raise ValueError("partitioning needs to be actual Partitioning object")
ValueError: partitioning needs to be actual Partitioning object
```
I didn't add real world use cases (or python bindings) because I figured those would be covered by #11008.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #11014: ARROW-13775: [C++] Allow Partitioning objects to be created with a vector of field names
Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #11014:
URL: https://github.com/apache/arrow/pull/11014#issuecomment-908595122
> What would be a use case? (I don't see any test where it is actually used for something) I suppose as alternative for #11008, right?
Yes, but not as an alternative, this would be to enable #11008. I don't believe #11008 works today. `ds.partitioning(field_names=['a'])` returns a partitioning factory and not a partitioning. It also returns an error when specifying the hive format. A partitioning factory cannot be used for writing datasets (only reading)...
```
>>> import pyarrow
>>> import pyarrow as pa
>>> import pyarrow.dataset as ds
>>> table = pa.Table.from_pydict({'a': [1, 2, 3], 'b': ['x', 'y', 'z']})
>>> ds.partitioning(field_names=['a'])
<pyarrow._dataset.PartitioningFactory object at 0x7f2410d4c170>
>>> part = ds.partitioning(field_names=['a'])
>>> ds.write_dataset(table, '/tmp/new_dataset', format='parquet', partitioning=part)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pace/anaconda3/envs/arrow-release-5/lib/python3.9/site-packages/pyarrow/dataset.py", line 791, in write_dataset
partitioning = _ensure_write_partitioning(partitioning)
File "/home/pace/anaconda3/envs/arrow-release-5/lib/python3.9/site-packages/pyarrow/dataset.py", line 686, in _ensure_write_partitioning
raise ValueError("partitioning needs to be actual Partitioning object")
ValueError: partitioning needs to be actual Partitioning object
```
I didn't add real world use cases (or python bindings) because I figured those would be covered by #11008.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org