[jira] [Created] (PROTON-668) Document Proton-c IO restrictions for 0.8 release

["Cliff Jansen (JIRA)" <ji...@apache.org>]

Cliff Jansen created PROTON-668:
-----------------------------------

             Summary: Document Proton-c IO restrictions for 0.8 release
                 Key: PROTON-668
                 URL: https://issues.apache.org/jira/browse/PROTON-668
             Project: Qpid Proton
          Issue Type: Improvement
          Components: proton-c
    Affects Versions: 0.8
            Reporter: Cliff Jansen
            Assignee: Cliff Jansen
            Priority: Blocker
             Fix For: 0.8


Proton is designed to provide an efficient IO layer that functions without imposing a threading model on the application.  Applications may (1) roll their own IO and just use the Proton engine, (2) use all Proton primitives, (3) use some Proton primitives augmented by an external event loop.

Case (1) is unrelated to this JIRA.  The others may be restated:

Scenario 2: Proton event loop: a proton selector manages socket events for all sockets placed in the selector, all associated sockets use pn_io_xxx() calls.  Sockets outside the selector are "unmanaged" and passed through to the OS socket function unchanged.

Scenario 3: Third party event loop (no proton selector involved), all sockets are treated as for "unmanaged" in scenario 2.

Scenario 4, 5...: Others to support?


The problem:

The Proton Posix pattern for efficient IO is:

  "tell me when your (OS) buffer is ready for io transfer (in or out)"

Whereas the normal Windows pattern is somewhat reversed (IO completion ports):

  "tell me when you are done transferring data (to or from) my (user space) buffer"


The current Windows IOCP implementation (PROTON-640) tries to make the latter look like the former with some constraints.   There should be documentation specifying reasonable limits on Proton usage that may be falsely implied by the API but do not translate efficiently to Windows.  Assuming that future Windows implementations may adopt more aggressive performance strategies (especially on the read side), I would propose something along the lines of:


  a socket may only ever be used with a single pn_io_t in its lifetime
    exception: a socket from pn_accept() is not yet associated with any pn_io_t and its first use can be with any pn_io_t (or never with a pn_io_t at all)

  send/recv/close may not be intermixed with similar non-Proton OS calls (otherwise: out of order or lost data)

  a socket can move once from an external loop to a proton loop, but never the other way

  pn_pipe() values can only be used with pn_read and pn_write and pn_selector_select, they cannot participate in an external event loop.

  Furthermore, there is no thread safety except:

    threads may do concurrent pn_io_xxx() calls as long as no two are simultaneous on the same socket (where xxx is send/recv/read/write)

    pn_selector_select() is thread safe against pn_read/pn_write/pn_send/pn_recv, but the outcome of the select is indeterminate.  pn_selector_select() must be interrupted and restarted at any time when other simultaneous IO may affect the outcome.

    calls on different pn_io_t objects do not interact and are thread safe.


If it is desirable for a socket to be used in an external loop after being used in a Proton loop, we would need some sort of blocking calls along the lines of:

  pn_io_flush()
  pn_io_drain()

which would be no-ops on Posix but would unwind outstanding completions on Windows.


Early criticism of any of the above assumptions would be greatly appreciated.  I will try to reword the above, or its evolution into the existing documentation for 0.8.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Created] (PROTON-668) Document Proton-c IO restrictions for 0.8 release

[Bozo Dragojevic <bo...@digiverse.si>]

Hi Cliff,

comments inline

On 11. 09. 14 09:32, Cliff Jansen (JIRA) wrote:
>
> Proton is designed to provide an efficient IO layer that functions without imposing a threading model on the application.  Applications may (1) roll their own IO and just use the Proton engine, (2) use all Proton primitives, (3) use some Proton primitives augmented by an external event loop.
>
> Case (1) is unrelated to this JIRA.  The others may be restated:
>
> Scenario 2: Proton event loop: a proton selector manages socket events for all sockets placed in the selector, all associated sockets use pn_io_xxx() calls.  Sockets outside the selector are "unmanaged" and passed through to the OS socket function unchanged.
>
> Scenario 3: Third party event loop (no proton selector involved), all sockets are treated as for "unmanaged" in scenario 2.

yay, looks like my use-case :)

> Scenario 4, 5...: Others to support?
>
>
> The problem:
>
> The Proton Posix pattern for efficient IO is:
>
>   "tell me when your (OS) buffer is ready for io transfer (in or out)"
>
> Whereas the normal Windows pattern is somewhat reversed (IO completion ports):
>
>   "tell me when you are done transferring data (to or from) my (user space) buffer"
>
> The current Windows IOCP implementation (PROTON-640) tries to make the latter look like the former with some constraints.   There should be documentation specifying reasonable limits on Proton usage that may be falsely implied by the API but do not translate efficiently to Windows.  Assuming that future Windows implementations may adopt more aggressive performance strategies (especially on the read side), I would propose something along the lines of:
>
>
>   a socket may only ever be used with a single pn_io_t in its lifetime
>     exception: a socket from pn_accept() is not yet associated with any pn_io_t and its first use can be with any pn_io_t (or never with a pn_io_t at all)

I'm not sure if it's implied, but pn_connect() and pn_listen() also need
to support 'third party event loop'.
Specifically, pn_connect() has to remain non-blocking (we get to know
about the connect error later in the external event loop)

>   send/recv/close may not be intermixed with similar non-Proton OS calls (otherwise: out of order or lost data)
>
>   a socket can move once from an external loop to a proton loop, but never the other way
>
>   pn_pipe() values can only be used with pn_read and pn_write and pn_selector_select, they cannot participate in an external event loop.
>

External event loops should have their own mechanism how to signal
non-socket events into the loop.

>   Furthermore, there is no thread safety except:
>
>     threads may do concurrent pn_io_xxx() calls as long as no two are simultaneous on the same socket (where xxx is send/recv/read/write)
>

This will break pn_io_error() and pn_io_wouldblock() as they are defined
now.

>     pn_selector_select() is thread safe against pn_read/pn_write/pn_send/pn_recv, but the outcome of the select is indeterminate.  pn_selector_select() must be interrupted and restarted at any time when other simultaneous IO may affect the outcome.
>

When you say 'interrupted' is there a simpler way than a pn_write() to
writeable pn_socket_t of pn_pipe()
 that has it's readable pn_socket_t associated with a pn_selectable_t
that is added to said pn_selector_t ? ;)

I have a feeling you don't really want/need to expose the pn_pipe(), but
add a
pn_selector_interrupt() and a mechanism of querying that for the caller
of pn_selector_select()
especially as you want to implement it completely differently on windows.

>     calls on different pn_io_t objects do not interact and are thread safe.
>
>
> If it is desirable for a socket to be used in an external loop after being used in a Proton loop, we would need some sort of blocking calls along the lines of:
>
>   pn_io_flush()
>   pn_io_drain()
>
> which would be no-ops on Posix but would unwind outstanding completions on Windows.
>
>

this part sounds ok, if it is needed

Thanks,
Bozzo