[jira] [Commented] (FLINK-8395) Create (derived) sliced Buffer class

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324016#comment-16324016 ] 

ASF GitHub Bot commented on FLINK-8395:
---------------------------------------

GitHub user NicoK opened a pull request:

    https://github.com/apache/flink/pull/5288

    [FLINK-8395][network] add a read-only sliced ByteBuf implementation based on NetworkBuffer

    ## What is the purpose of the change
    
    To insert the same buffer (or sub-regions from it) multiple times into Netty's queues, we also need a (read-only) sliced variant of our `NetworkBuffer` that implements our `Buffer` interface. This PR adds that implementation and is based on #4613 (included here) and required by #4615.
    
    Please note that this has been reviewed in dataArtisans/flink#5
    
    ## Brief change log
    
    - add `ReadOnlySlicedNetworkBuffer`
    - adapt `NetworkBuffer` to be able to generate these sliced buffers
    - add `ReadOnlySlicedBufferTest` that performs tests for the `Buffer` interface's wrapper methods
    
    ## Verifying this change
    
    This change added tests and can be verified as follows:
    
      - added `ReadOnlySlicedBufferTest` (see above)
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): **no**
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
      - The serializers: **no**
      - The runtime per-record code paths (performance sensitive): **no** (per buffer only)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
      - The S3 file system connector: **no**
    
    ## Documentation
    
      - Does this pull request introduce a new feature? **no**
      - If yes, how is the feature documented? **JavaDocs**


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/NicoK/flink flink-8395

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5288.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5288
    
----
commit b636592f237afb745e444bed5d0520c5736015b4
Author: Nico Kruber <ni...@...>
Date:   2017-08-31T10:46:01Z

    [hotfix][tests] replace DiscardingRecycler with FreeingBufferRecycler

commit 0761351db2c975ea02718f725e667b52dd9778a4
Author: Nico Kruber <ni...@...>
Date:   2017-12-12T14:05:14Z

    [hotfix][tests] do not use a mocked BufferRecycler for unpooled memory segments
    
    The mock will actually keep references to the segments instead of freeing them.

commit 63832c24d22e672239389550e377661d1715a27c
Author: Nico Kruber <ni...@...>
Date:   2018-01-03T14:57:47Z

    [hotfix][tests] make SpillableSubpartitionTest use TestBufferFactory.createBuffer
    
    (this simplifies the test setups)

commit 070a314648b319839c2049d5c557b7608f5ce8ae
Author: Nico Kruber <ni...@...>
Date:   2018-01-08T10:48:25Z

    [hotfix][tests] replace InputChannelTestUtils#createMockBuffer() with TestBufferFactory#createBuffer()
    
    This eliminates one more unnecessary buffer mock.

commit 09d73527f90f48838631edb72d806a5bfd7d8b69
Author: Nico Kruber <ni...@...>
Date:   2018-01-09T16:08:50Z

    [FLINK-7520][network] let our Buffer class extend from netty's buffer class
    
    For this, use a common (flink) Buffer interface and an implementation
    (NetworkBuffer) that implements netty's buffer methods as well. In the future,
    with this, we are able to avoid unnecessary buffer copies when handing buffers
    over to netty while keeping our MemorySegment logic and configuration.
    
    For the netty-specific part, the NetworkBuffer also requires a ByteBuf allocator
    which is otherwise not needed in our use cases, so if the buffer is handed over
    to netty, it requires a byte buffer allocator to be set.

commit 4fbef3c0c64750f6c38bafaa850d670392b0ecc8
Author: Nico Kruber <ni...@...>
Date:   2018-01-08T16:02:03Z

    [hotfix][io] remove duplicate code between SynchronousBufferFileReader and BufferReadRequest

commit 7a1b4031202861fbdb3d8ae86f3810c36761b1e3
Author: Nico Kruber <ni...@...>
Date:   2018-01-11T11:03:45Z

    [hotfix][network] clarify BufferResponse#size() uses (by removing it)
    
    This field was only used by the code paths on the receiver and was inconsistent
    with what was added on the sending side. We should use the contained buffer's
    readableBytes() instead, depending on the actual use case.

commit 2dad095f38e7ccd31cb5e12145848909b79d02d0
Author: Nico Kruber <ni...@...>
Date:   2018-01-09T21:56:38Z

    [hotfix][network] rename Buffer#retain() and #recycle in preparation for FLINK-8396 and FLINK-8395
    
    Since these two methods also exist in Netty's ByteBuf, we would otherwise get
    into overloading conflicts.
    
    Also add Buffer#readableBytes() and Buffer#setAllocator().

commit e2fa9ccfed01bfdd756918116ccff20ed1833537
Author: Nico Kruber <ni...@...>
Date:   2018-01-09T21:57:18Z

    [FLINK-8395][network] add a read-only sliced ByteBuf implementation based on NetworkBuffer

----


> Create (derived) sliced Buffer class
> ------------------------------------
>
>                 Key: FLINK-8395
>                 URL: https://issues.apache.org/jira/browse/FLINK-8395
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Network
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
>
> In order to pass sub-regions of a single buffer separately, we require a sliced Buffer instance with a shared {{MemorySegment}} and reference counting but separate indices. This should be read-only.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)