You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/09/22 15:35:15 UTC

[GitHub] [arrow] LouisClt opened a new issue, #14211: Possibility to write ORC by chunks (C++)

LouisClt opened a new issue, #14211:
URL: https://github.com/apache/arrow/issues/14211

   Hello,
   I use Arrow as an intermediate format before writing Parquet or ORC.
   However, in order to reduce the memory fottprint (as the tables can be quite big), I need to write both formats little by little.
   For the parquet export, I manager to do so by using the "WriteColumnChunk" method.
   However, for the ORC format, there is no such thing, the only method available is "Write(table)" which writes a whole table into the file. That forces me to allocate a whole Arrow table in memory beforehand, which I can't do.
   So is there a plan to support this in the future ?
   
   I looked at the code, and it seems to be pretty easy to make a method like that, since the "write" method already write little chunks internally. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] wjones127 commented on issue #14211: Possibility to write ORC by chunks (C++)

Posted by GitBox <gi...@apache.org>.
wjones127 commented on issue #14211:
URL: https://github.com/apache/arrow/issues/14211#issuecomment-1255215386

   Looking at the ORC API, I don't think there's any requirement that you only call `Write()` once; so you can at least write in batches (call `Write()` on each batch of data, then `Close()`) at the end.
   
   I don't think there is an API to write a column at a time. If you would like, you are welcome to open an issue on [our Jira](https://issues.apache.org/jira/projects/ARROW/issues/) to discuss the possibility.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] wjones127 commented on issue #14211: Possibility to write ORC by chunks (C++)

Posted by GitBox <gi...@apache.org>.
wjones127 commented on issue #14211:
URL: https://github.com/apache/arrow/issues/14211#issuecomment-1255267638

   It's also possible the API is misleading; I'm not super familiar with the implementation. It doesn't seem like the internal writer is actually re-used...
   
   https://github.com/apache/arrow/blob/5256e434935f90f02023e6dfce3b5c1ed056a1c9/cpp/src/arrow/adapters/orc/adapter.cc#L733-L734


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] LouisClt commented on issue #14211: Possibility to write ORC by chunks (C++)

Posted by GitBox <gi...@apache.org>.
LouisClt commented on issue #14211:
URL: https://github.com/apache/arrow/issues/14211#issuecomment-1255233360

   Thanks, for your answer. I tried the approach you are suggesting, but I could not get it to work, it failed when I tried to reimport it afterwards. But maybe that was a mistake in my testing, so I will give it another try.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] LouisClt commented on issue #14211: Possibility to write ORC by chunks (C++)

Posted by GitBox <gi...@apache.org>.
LouisClt commented on issue #14211:
URL: https://github.com/apache/arrow/issues/14211#issuecomment-1255935235

   I re-tested it and indeed, the following import was not working. 
   But it indeed gave me the idea to try to re-use the "writer_" rather than reallocating it everytime.
   
   Doing so, it was finally working. I think I will do a PR for this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] LouisClt closed issue #14211: Possibility to write ORC by chunks (C++)

Posted by GitBox <gi...@apache.org>.
LouisClt closed issue #14211: Possibility to write ORC by chunks (C++)
URL: https://github.com/apache/arrow/issues/14211


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org