[GitHub] [arrow] EB80 commented on issue #11665: read_feather error with 32 GB file

[GitBox <gi...@apache.org>]

EB80 commented on issue #11665:
URL: https://github.com/apache/arrow/issues/11665#issuecomment-969113711


   Thanks for the reply,
   
   Additional information:
   -The feather was written with feather::write_feather and not arrow::write_feather
   -I receive the same error with older versions of arrow (currently running 6.0.0.2)
   -I actually encounter a separate issue attempting to write with arrow::write_feather.  78MB of the 32GB file will write, and then the writing hangs indefinitely without continuing or crashing.  (I used feather::write_feather for this reason.)
   -I am writing to a network drive and not a local disk
   -The dataframe is 26M rows and 150 columns
   
   Traceback:
   6: stop(e)
   5: value[[3L]](cond)
   4: tryCatchOne(expr, names, parentenv, handlers[[1L]])
   3: tryCatchList(expr, classes, parentenv, handlers)
   2: tryCatch(reader$Read(columns), error = read_compressed_error)
   1: arrow::read_feather("Archive/2021-06-23/2021-06-23 - Pre-Processed DECKPLATE - June 2021.feather")
   
   Thank you,
   Edward
   
   Edward Beall, PhD
   Institute for Defense Analyses
   Operational Evaluation Division
   Office: (703)845-2463
   Cell: (443)277-2648
   ***@***.******@***.***>
   ***@***.******@***.***>
   
   From: Joris Van den Bossche ***@***.***>
   Sent: Monday, November 15, 2021 8:28 AM
   To: apache/arrow ***@***.***>
   Cc: Beall, Edward J ***@***.***>; Mention ***@***.***>
   Subject: [EXT] Re: [apache/arrow] read_feather error with 32 GB file (Issue #11665)
   
   *** This email originated outside of IDA. Please verify that you recognize the sender and know the content is safe before proceeding. ***
   
   
   
   @EB80<https://github.com/EB80> Thanks for the report!
   
   Would you be able to share some more info? (for example the full traceback, how you created the file, whether you can read it with an older version, and if so what's the schema is, ..?) Or if possible provide a reproducible example?
   
   —
   You are receiving this because you were mentioned.
   Reply to this email directly, view it on GitHub<https://github.com/apache/arrow/issues/11665#issuecomment-968909862>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AWNWW7752TVCITQOBYR3XNDUMEDEJANCNFSM5HYQCQ7Q>.
   Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org