You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/11/10 17:48:53 UTC
[GitHub] [arrow] EB80 opened a new issue #11665: read_feather error with 32 GB file
EB80 opened a new issue #11665:
URL: https://github.com/apache/arrow/issues/11665
arrow v.6.0.0.2 results in the following error when attempting read_feather on a 32 GB feather file:
Error: Invalid: Invalid read (offset = 7140512496, size = -956703880)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
thisisnic commented on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-1009027381
@romainfrancois Jon suggested I ask you if there is any batching happening in the R data frame to Arrow table conversion? If not, that could explain what's happening here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] rockhowse commented on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
rockhowse commented on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-1006965295
Did some extensive testing of this scenario on both Windows 10 (32GB and 128GB RAM machines) as well as MacOSX (32GB) machine. I tested all using `R 4.1.2` and `arrow 6.0.1` as those are the latest stable versions of R and the arrow module from CRAN. The most consistent producer of the scenario was the Windows 10 64GB machine using default Page File configuration.
All testing steps, observations and screenshots of RStudio + Memory/SWP consumption while testing I popped into this github repo:
https://github.com/rockhowse/apache_arrow_32GB_write_feather
Prime example of the state you can get into even with 128GB of RAM and the provided R code is shown on [this screenshot](https://github.com/rockhowse/apache_arrow_32GB_write_feather/blob/main/windows_10_21H2_128GB_RAM_2TB_nVME/screenshots/004_01_feather_128GB_RAM_hang_0_CPU_0_disk.PNG).
Here's what I see occurring:
- DF is created successfully, gobbling up between 32-64GB of RAM
- fwrite completes successfully with a ~44GB file written out peaking out at/around ~68-70GB of RAM used
- feather write steadily increases in RAM in a period with no disk activity (memcpy?) even hitting 100% on the 128GB box
- At this point it can go two ways:
- completes successfully with ~14GB feather file
- gets into a state after hitting ~100% RAM consumption where CPU and disk usage drop to 0% and the application "hangs" without error
- If it get into a hung state, you have to force kill RStudio, [I have gotten dialogs](https://github.com/rockhowse/apache_arrow_32GB_write_feather/blob/main/windows_10_21H2_128GB_RAM_2TB_nVME/screenshots/011_03_term_RStudio_fails_to_respond.PNG) stating that the app isn't responding to the attempt to close it and have to do a force.
Interestingly enough... when testing the same versions of `R` and `arrow` on MacOSX with a 32GB box it does in fact complete, but takes forever due to MacOSX having to constantly resize the SWP partition to accommodate the increase requirement for virtual memory. It peaks out at/around 32GB of SWP.
I haven't yet re-configured my windows developer environment on my new rig, but will set it up and see what the debugger has to say. My gut feeling is that we are in a `DOM` vs `SAX` type scenario where the feather write implementation as it's currently being consumed in `R` is doing a lot more memory consumption than might be obvious with smaller data sets. Given there is a large period of time where there is no disk IO but lots of CPU and memory it would lead me to believe it's not "streaming" as effectively as it could be.
Hopefully this is helpful. I used this issue as a way for me to get back up to speed with arrow as it's been a couple years. I think I will start by using R debugging as suggested by @thisisnic. I am not well versed in lower level debugging in R, but if I can hook into a debug build of the native arrow lib being used, it should open up a bit more visibility into what's going on under the hood during that call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-975557533
@EB80 Is there any way you can share the code to produce the file (or a similar file that would fail loading with `arrow::read_feather` but succeed with `feather::read_feather`, so that we can try to reproduce?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] EB80 commented on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
EB80 commented on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-969113711
Thanks for the reply,
Additional information:
-The feather was written with feather::write_feather and not arrow::write_feather
-I receive the same error with older versions of arrow (currently running 6.0.0.2)
-I actually encounter a separate issue attempting to write with arrow::write_feather. 78MB of the 32GB file will write, and then the writing hangs indefinitely without continuing or crashing. (I used feather::write_feather for this reason.)
-I am writing to a network drive and not a local disk
-The dataframe is 26M rows and 150 columns
Traceback:
6: stop(e)
5: value[[3L]](cond)
4: tryCatchOne(expr, names, parentenv, handlers[[1L]])
3: tryCatchList(expr, classes, parentenv, handlers)
2: tryCatch(reader$Read(columns), error = read_compressed_error)
1: arrow::read_feather("Archive/2021-06-23/2021-06-23 - Pre-Processed DECKPLATE - June 2021.feather")
Thank you,
Edward
Edward Beall, PhD
Institute for Defense Analyses
Operational Evaluation Division
Office: (703)845-2463
Cell: (443)277-2648
***@***.******@***.***>
***@***.******@***.***>
From: Joris Van den Bossche ***@***.***>
Sent: Monday, November 15, 2021 8:28 AM
To: apache/arrow ***@***.***>
Cc: Beall, Edward J ***@***.***>; Mention ***@***.***>
Subject: [EXT] Re: [apache/arrow] read_feather error with 32 GB file (Issue #11665)
*** This email originated outside of IDA. Please verify that you recognize the sender and know the content is safe before proceeding. ***
@EB80<https://github.com/EB80> Thanks for the report!
Would you be able to share some more info? (for example the full traceback, how you created the file, whether you can read it with an older version, and if so what's the schema is, ..?) Or if possible provide a reproducible example?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/apache/arrow/issues/11665#issuecomment-968909862>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AWNWW7752TVCITQOBYR3XNDUMEDEJANCNFSM5HYQCQ7Q>.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] EB80 commented on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
EB80 commented on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-972958931
Thanks all-
I read the file in with feather::read_feather, and there was not an issue. I'm assuming this was the problem with arrow::read_feather (i.e., the file was written with feather vice arrow), but now we're back to the writing error.
When I attempt to write the dataframe with arrow::write_feather, there is not disk activity once it encounters the indefinite hanging. There is a ton of memory usage, but that's expected with a huge dataframe in the RStudio workspace. I excluded the possibility of some sort of an issue with the dataframe itself by writing successfully with data.table::fwrite.
Regarding dbg, the MinGW download failed for some reason, and I haven't figured out an alternative way to debug via the Windows Command Prompt. (Help for a simpleton would be appreciated.)
Interestingly, while the arrow::write_feather has hung every time so far, it stopped at 156MB one time and 78MB every other time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-971409478
Aother suggestion: if it was written with the `feather` package, could you try reading the file with that package? And if that works, write the file again using `arrow::write_feather`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
thisisnic commented on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-971428232
In addition to `pitrou`'s comment on retrying `arrow::write_feather()` and checking the CPU activity, if you like, you could generate more diagnostic information by running the code with the C++ debugger attached.
If you want to do this, there are some instructions in the dev version of the docs (https://ursalabs.org/arrow-r-nightly/articles/developers/debugging.html) but here's a short version:
1. Start up R in a terminal with the debugger using either `R -d gdb` or `R -d lldb` - hopefully one of these should work with your OS
2. Type `run` to run R
3. Run the code you run which causes the session to hang
At this point, there should either be a load of extra output detailing the cause of the problem or the session will just hang again. If it just hangs, press Crtl+C to stop the debugger and then type in `thread apply all bt` - this will generate a tonne of output but it'll be really useful stuff for us!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou edited a comment on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
pitrou edited a comment on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-975557533
@EB80 Is there any way you can share the code to produce the file (or a similar file that would fail loading with `arrow::read_feather` but succeed with `feather::read_feather`), so that we can try to reproduce?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] EB80 edited a comment on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
EB80 edited a comment on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-969113711
Thanks for the reply,
Additional information:
-The feather was written with feather::write_feather and not arrow::write_feather
-I receive the same error with older versions of arrow (currently running 6.0.0.2)
-I actually encounter a separate issue attempting to write with arrow::write_feather. 78MB of the 32GB file will write, and then the writing hangs indefinitely without continuing or crashing. (I used feather::write_feather for this reason.)
-I am writing to a network drive and not a local disk
-The dataframe is 26M rows and 150 columns
Traceback:
6: stop(e)
5: value[[3L]](cond)
4: tryCatchOne(expr, names, parentenv, handlers[[1L]])
3: tryCatchList(expr, classes, parentenv, handlers)
2: tryCatch(reader$Read(columns), error = read_compressed_error)
1: arrow::read_feather("Archive/2021-06-23/2021-06-23 - Pre-Processed DECKPLATE - June 2021.feather")
Thank you,
Edward
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-968909862
@EB80 Thanks for the report!
Would you be able to share some more info? (for example the full traceback, how you created the file, whether you can read it with an older version, and if so what's the schema is, ..?) Or if possible provide a reproducible example?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] EB80 edited a comment on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
EB80 edited a comment on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-975690421
I expect that the issue with arrow::read_feather was just because I had used the very old feather::write_feather to write the file.
I have the following code to test arrow::write_feather:
```R
rm(list = ls())
# set the wd to be where this script is saved
setwd(dirname(rstudioapi::getActiveDocumentContext()$path))
# set dimensions based on the real file
numRows = 26e6 # 26M rows in the real file
numCols = 150 # 150 columns in the real file
# whip up a fake dataframe
fakeDataframe <- as.data.frame(matrix("fake string", numRows, numCols))
# change the column names for aesthetic purposes, I guess
names(fakeDataframe) <- sprintf("Fake Column %s", 1:150)
# save the fake file with data.table
data.table::fwrite(fakeDataframe, "fakeFile.csv")
# save the fake file with arrow
arrow::write_feather(fakeDataframe, "fakeFile.feather")
```
The fwrite step took about 10 minutes to write. While the dimensions of the fake file match those of the real file, the size on the disk is much larger (46 GB vice 32 GB). I wrote a while loop to trim off rows from the fake file until it matched the real file, but object.size() was painfully slow. Either way, I figured this would be a suitable test. arrow::write_feather hangs with this fake dataframe just as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] EB80 commented on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
EB80 commented on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-975690421
I expect that the issue with arrow::read_feather was just because I had used the very old feather::write_feather to write the file.
I have the following code to test arrow::write_feather:
"
rm(list = ls())
# set the wd to be where this script is saved
setwd(dirname(rstudioapi::getActiveDocumentContext()$path))
# set dimensions based on the real file
numRows = 26e6 # 26M rows in the real file
numCols = 150 # 150 columns in the real file
# whip up a fake dataframe
fakeDataframe <- as.data.frame(matrix("fake string", numRows, numCols))
# change the column names for aesthetic purposes, I guess
names(fakeDataframe) <- sprintf("Fake Column %s", 1:150)
# save the fake file with data.table
data.table::fwrite(fakeDataframe, "fakeFile.csv")
# save the fake file with arrow
arrow::write_feather(fakeDataframe, "fakeFile.feather")
"
The fwrite step took about 10 minutes to write. While the dimensions of the fake file match those of the real file, the size on the disk is much larger (46 GB vice 32 GB). I wrote a while loop to trim off rows from the fake file until it matched the real file, but object.size() was painfully slow. Either way, I figured this would be a suitable test. arrow::write_feather hangs with this fake dataframe just as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-971406320
> The feather was written with feather::write_feather and not arrow::write_feather
The standalone `feather` package is an extremely old package that is not supported anymore, so I would not recommend using it.
> 78MB of the 32GB file will write, and then the writing hangs indefinitely without continuing or crashing
Is the CPU busy when doing this or idle? Do you see any disk activity?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
wesm commented on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-980795569
I recall a number of issues writing data frames over 4GB with the old `feather` package, there were some other issues in the past, and there may be some existing Jira issues, too.
https://github.com/wesm/feather/issues/380
https://github.com/wesm/feather/issues/372
It would be good to get to the bottom of the various issues here at least from the `arrow::` package side. I have no idea why writing a FileOutputStream to a network drive would hang (which is what sounds like is happening)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #11665: read_feather error with 32 GB file
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-976290127
cc @wesm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org