You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "DDtKey (via GitHub)" <gi...@apache.org> on 2023/08/28 22:52:49 UTC
[GitHub] [arrow-datafusion] DDtKey opened a new issue, #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
DDtKey opened a new issue, #7436:
URL: https://github.com/apache/arrow-datafusion/issues/7436
### Describe the bug
Methods `write_csv`, `write_parquet`, `write_json` and etc have been broken after switching to `object-store`.
They produce files with wrong extension with additional suffix, for example: `filename_4.csv#1`
The name seems to change [here](https://github.com/apache/arrow-rs/blob/cb793a5d54e155b121bc5c33534a37bf94651c84/object_store/src/local.rs#L658) and is not controlled directly by datafusion.
I think this is more an issue of `object_store` for local file-system implementation, but the behavior have been broken for datafusion after switching so I raised the issue.
### To Reproduce
```rs
let ctx = SessionContext::new();
// register any sources
let data_frame = ctx.sql("...").await?; // any valid SQL
data_frame.write_csv().await?;
```
### Expected behavior
Filenames should be with correct extension, without suffix of `#1`.
`write_{format}` methods used to create file per thread (currently it's controlled by `per_thread_output: true`) with correct names and an index: `{filename}_{index}.{ext}`.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] DDtKey commented on issue #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
Posted by "DDtKey (via GitHub)" <gi...@apache.org>.
DDtKey commented on issue #7436:
URL: https://github.com/apache/arrow-datafusion/issues/7436#issuecomment-1697643304
That's definitely right point to avoid staged files.
I think `ObjectStore` should create staged files as a temporary files (probably just hidden ones) with `Drop` logic to delete them in case they weren't completed.
I don't think DF should care of the way `object_store` handles the staging. If a file write/upload wasn't completed - intermediate files should be cleaned up. Probably this may be configurable behavior
WDYT?
cc @tustvold @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] DDtKey commented on issue #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
Posted by "DDtKey (via GitHub)" <gi...@apache.org>.
DDtKey commented on issue #7436:
URL: https://github.com/apache/arrow-datafusion/issues/7436#issuecomment-1697080314
Sorry, my bad. Seems it works as expected
Looks like it happens in case of:
- error in one of threads which interrupts the execution of others (when files weren't fully written to disk)
- stopping of the runtime
I.e when rename call by `object_store` wasn't completed (https://github.com/apache/arrow-rs/blob/cb793a5d54e155b121bc5c33534a37bf94651c84/object_store/src/local.rs#L8111)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold commented on issue #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7436:
URL: https://github.com/apache/arrow-datafusion/issues/7436#issuecomment-1707106729
I created https://github.com/apache/arrow-rs/issues/4778 to track although I'm not sure how actually sure how to implement it, as Rust futures don't have an async Drop facility...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] DDtKey commented on issue #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
Posted by "DDtKey (via GitHub)" <gi...@apache.org>.
DDtKey commented on issue #7436:
URL: https://github.com/apache/arrow-datafusion/issues/7436#issuecomment-1697742137
Yeah, that's true
Perhaps a wrapper with `Drop` (with remove file call) is an option? as a fallback, it can of course fail and is not guaranteed, but at least it covers typical flows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] devinjdangelo commented on issue #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #7436:
URL: https://github.com/apache/arrow-datafusion/issues/7436#issuecomment-1697620779
I think there may still be an issue here with the behavior you are seeing. When one thread encounters an error, no files should be written to disk, so we may want to add some logic to clean up the staged local files (unclear to me if that logic should be in the ObjectStore crate or DF at a glance).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] DDtKey commented on issue #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
Posted by "DDtKey (via GitHub)" <gi...@apache.org>.
DDtKey commented on issue #7436:
URL: https://github.com/apache/arrow-datafusion/issues/7436#issuecomment-1696963350
I encountered it exactly against '30.0.0' after updating from '28.0.0'.
Let me provide some reproducible example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold commented on issue #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7436:
URL: https://github.com/apache/arrow-datafusion/issues/7436#issuecomment-1697662514
Yes ideally we would stage the files in such a way that they get automatically cleaned up if the operation aborts. On Unix platforms this is fairly straightforward as there is the unlink command, it gets a bit mess on Windows though... I can add it to my list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7436:
URL: https://github.com/apache/arrow-datafusion/issues/7436#issuecomment-1707044566
Would it be good to file a ticket in arrow-rs / obect-store for this behavior (even if it is non trivial to implement)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] devinjdangelo commented on issue #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #7436:
URL: https://github.com/apache/arrow-datafusion/issues/7436#issuecomment-1696602551
Which version of DataFusion are you using? I was not able to replicate this behavior using the current `main` branch or using v30.0.0. I
If you are using DataFusion version prior to 30.0.0, try that and see if it is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] DDtKey closed issue #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
Posted by "DDtKey (via GitHub)" <gi...@apache.org>.
DDtKey closed issue #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
URL: https://github.com/apache/arrow-datafusion/issues/7436
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] DDtKey closed issue #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
Posted by "DDtKey (via GitHub)" <gi...@apache.org>.
DDtKey closed issue #7436: Broken behavior: `write_{format}` methods crate wrong csv files in local FS
URL: https://github.com/apache/arrow-datafusion/issues/7436
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org