You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "DavidEscott (via GitHub)" <gi...@apache.org> on 2024/01/18 22:04:27 UTC

[I] dataset.write_dataset needs a better API for append operations [arrow]

DavidEscott opened a new issue, #39693:
URL: https://github.com/apache/arrow/issues/39693

### Describe the enhancement requested

The documentation on write_dataset says ‘overwrite_or_ignore’ combined with a unique basename_template will allow an append workflow. I really struggle to make sense of how to do this in a reasonably effective manner.

Suppose my dataset is partitioned by month, and I want to append some data that spans multiple months. It will be pyarrow's responsibility to split the data into two subsets written to two different folders, but it is my responsibility to generate a basename_template that is unique across these two folders.

I believe the expected resolution is that the caller generates a UUID and calls the function with `basename_template=uuid.uuid4() + "-{i}."`. While their are a couple other approaches that could be taken (e.g. use a datestamp or timestamp instead of a UUID), I generally prefer more opinionated APIs and would just ask for a way to directly ask pyarrow to append and have it handle the matter in whatever way it deems best.

Alternately there might be something in the dataset that itself might be useful to generate a unique basename. For instance if the data is partitioned by year/month perhaps the desired basename_template might be `{day}-{i}.parquet` and the behavior should be to "append data if the year-month-day is not already written to disk, otherwise ignore". In this situation I cannot provide a single unique `basename_template` for all the data I might desire to conditionally append. What I need is a callback that would build the basename_template for each subset after partition splitting.

### Component(s)

C++, Python

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] dataset.write_dataset needs a better API for append operations [arrow]

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.

thisisnic commented on issue #39693:
URL: https://github.com/apache/arrow/issues/39693#issuecomment-3921321094

   This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] dataset.write_dataset needs a better API for append operations [arrow]

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.

thisisnic closed issue #39693: [Python] dataset.write_dataset needs a better API for append operations
URL: https://github.com/apache/arrow/issues/39693


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] dataset.write_dataset needs a better API for append operations [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on issue #39693:
URL: https://github.com/apache/arrow/issues/39693#issuecomment-3547085593

   This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] dataset.write_dataset needs a better API for append operations [arrow]

Posted by "davlee1972 (via GitHub)" <gi...@apache.org>.

davlee1972 commented on issue #39693:
URL: https://github.com/apache/arrow/issues/39693#issuecomment-1904976932

   I opened up this ticket to improve basename templates..
   https://github.com/apache/arrow/issues/39447


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org