You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Micah Kornfield (Jira)" <ji...@apache.org> on 2021/11/05 23:33:00 UTC
[jira] [Commented] (ARROW-11901) [Java] Investigate potential
performance improvement of compression codec
[ https://issues.apache.org/jira/browse/ARROW-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17439561#comment-17439561 ]
Micah Kornfield commented on ARROW-11901:
-----------------------------------------
{quote} As Samuel pointed out, it might be a valid idea to base the Java API on JavaCPP, but this is not the right place for this discussion (a thread in the mailing list?).
{quote}
This would be a mailing dev@ mailing list discussion. I don't think we would eliminate the existing API, but there might be some interest alternative Java APIs.
{quote}Seeing where the JavaCPP is used I think it is a viable project. I could contribute my {{CompressionCodec}} implementation to Arrow if this is desired. Creating JNI bindings for LZ4 in the Arrow repository would take more time and I won't be able to do this soon.
{quote}
[~benjamin.wilhelm@knime.com] Do you have pointers? I looked maybe too quickly and didn't see it used in other Apache projects for instance. If you have something that works for your use-case that is great, and if you want to open-source it also great, but it might need to live in a KNIME hosted project for the time being. I believe Arrow is now building JNI bindings for all major platforms, so the release story is a little bit better for a JNI code hosted by Arrow, I'll see how hard it would be to make the bindings at this point.
> [Java] Investigate potential performance improvement of compression codec
> -------------------------------------------------------------------------
>
> Key: ARROW-11901
> URL: https://issues.apache.org/jira/browse/ARROW-11901
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Java
> Reporter: Liya Fan
> Assignee: Benjamin Wilhelm
> Priority: Major
>
> In response to the discussion in https://github.com/apache/arrow/pull/8949/files#r588046787
> There are some performance penalties in the implementation of the compression codecs (e.g. data copying between heap/off-heap data). We need to revise the code to improve the performance.
> We should also provide some benchmarks to validate that the performance actually improves.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)