[jira] [Commented] (ARROW-10695) [C++][Dataset] Allow to use a UUID in the basename_template when writing a dataset

["Lance Dacey (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-10695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306996#comment-17306996 ] 

Lance Dacey commented on ARROW-10695:
-------------------------------------

Sorry, did not see a notification for this. Hm - I am not sure how to provide a minimal example easily. The issue is when multiple machines are writing to the same dataset at the same time into the same partition. 

For example, machine A downloads data from server 1 and saves it to the dataset at the same time as machine B downloading data and saving data from server 2.

My workaround for now was to  ensure that the basename_template is a unique value. Initially, I was using a UUID filename as the basename_template, but I need to be able to use fs.glob() to get a list of all of the fragments which were just written to process them in downstream tasks. Unfortunately, there is no metadata_collector for ds.write_dataset() yet.

> [C++][Dataset] Allow to use a UUID in the basename_template when writing a dataset
> ----------------------------------------------------------------------------------
>
>                 Key: ARROW-10695
>                 URL: https://issues.apache.org/jira/browse/ARROW-10695
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++
>            Reporter: Joris Van den Bossche
>            Priority: Minor
>              Labels: dataset, dataset-parquet-write
>             Fix For: 5.0.0
>
>
> Currently we allow the user to specify a {{basename_template}}, and this can include a {{"\{i\}"}} part to replace it with an automatically incremented integer (so each generated file written to a single partition is unique):
> https://github.com/apache/arrow/blob/master/python/pyarrow/dataset.py#L713-L717
> It _might_ be useful to also have the ability to use a UUID, to ensure the file is unique in general (not only for a single write) and to mimic the behaviour of the old {{write_to_dataset}} implementation.
> For example, we could look for a {{"\{uuid\}"}} in the template string, and if present replace it for each file with a new UUID.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)