You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by Chen Luo <cl...@uci.edu> on 2018/04/02 17:27:30 UTC

"Erroneous" Sync Write Mode in ResultState

Hi Devs,

As I saw last week, using the "sync" write mode to write data (i.e., in
Transaction Log Flusher) can lead to bad write throughput if the write
buffer is small. However, I just saw "sync" write mode is in ResultState
when we store the query result [1]. *Is this "sync" mode really needed on a
frame-basis*? The frame size is typically 32KB, and this could lead to low
throughput if a user wants to pull a lot of data out of our system. If
there is no specific need for this "sync" write, I'll submit a change to
fix it. Thanks!

[1]
https://github.com/apache/asterixdb/blob/77c8c79077ec33fae5944d982a337b784608aa87/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/dataset/ResultState.java#L130

Best regards,
Chen Luo

Re: "Erroneous" Sync Write Mode in ResultState

Posted by Murtadha Hubail <hu...@gmail.com>.
Hi Chen,

Please check if this won't cause any issues to asynchronous readers since they can read a file before it is closed, or no frames are written to the file yet.
If you don't find any issues there, please feel free to propose the change.

Regards,
Murtadha

On 04/02/2018, 9:45 PM, "Mike Carey" <dt...@gmail.com> wrote:

    +1 for the proposed relaxation...
    
    
    On 4/2/18 10:55 AM, Chen Luo wrote:
    > This is introduced by a recent fix [1]. It seems forcing writes when
    > ResultState is being closed should give same semantics (but with much
    > higher throughput)?
    >
    >
    > [1] https://asterix-gerrit.ics.uci.edu/#/c/2319/
    >
    > On Mon, Apr 2, 2018 at 10:27 AM, Chen Luo <cl...@uci.edu> wrote:
    >
    >> Hi Devs,
    >>
    >> As I saw last week, using the "sync" write mode to write data (i.e., in
    >> Transaction Log Flusher) can lead to bad write throughput if the write
    >> buffer is small. However, I just saw "sync" write mode is in ResultState
    >> when we store the query result [1]. *Is this "sync" mode really needed on
    >> a frame-basis*? The frame size is typically 32KB, and this could lead to
    >> low throughput if a user wants to pull a lot of data out of our system. If
    >> there is no specific need for this "sync" write, I'll submit a change to
    >> fix it. Thanks!
    >>
    >> [1] https://github.com/apache/asterixdb/blob/
    >> 77c8c79077ec33fae5944d982a337b784608aa87/hyracks-fullstack/
    >> hyracks/hyracks-control/hyracks-control-nc/src/main/
    >> java/org/apache/hyracks/control/nc/dataset/ResultState.java#L130
    >>
    >> Best regards,
    >> Chen Luo
    >>

Re: "Erroneous" Sync Write Mode in ResultState

Posted by Mike Carey <dt...@gmail.com>.
+1 for the proposed relaxation...


On 4/2/18 10:55 AM, Chen Luo wrote:
> This is introduced by a recent fix [1]. It seems forcing writes when
> ResultState is being closed should give same semantics (but with much
> higher throughput)?
>
>
> [1] https://asterix-gerrit.ics.uci.edu/#/c/2319/
>
> On Mon, Apr 2, 2018 at 10:27 AM, Chen Luo <cl...@uci.edu> wrote:
>
>> Hi Devs,
>>
>> As I saw last week, using the "sync" write mode to write data (i.e., in
>> Transaction Log Flusher) can lead to bad write throughput if the write
>> buffer is small. However, I just saw "sync" write mode is in ResultState
>> when we store the query result [1]. *Is this "sync" mode really needed on
>> a frame-basis*? The frame size is typically 32KB, and this could lead to
>> low throughput if a user wants to pull a lot of data out of our system. If
>> there is no specific need for this "sync" write, I'll submit a change to
>> fix it. Thanks!
>>
>> [1] https://github.com/apache/asterixdb/blob/
>> 77c8c79077ec33fae5944d982a337b784608aa87/hyracks-fullstack/
>> hyracks/hyracks-control/hyracks-control-nc/src/main/
>> java/org/apache/hyracks/control/nc/dataset/ResultState.java#L130
>>
>> Best regards,
>> Chen Luo
>>

Re: "Erroneous" Sync Write Mode in ResultState

Posted by Chen Luo <cl...@uci.edu>.
This is introduced by a recent fix [1]. It seems forcing writes when
ResultState is being closed should give same semantics (but with much
higher throughput)?


[1] https://asterix-gerrit.ics.uci.edu/#/c/2319/

On Mon, Apr 2, 2018 at 10:27 AM, Chen Luo <cl...@uci.edu> wrote:

> Hi Devs,
>
> As I saw last week, using the "sync" write mode to write data (i.e., in
> Transaction Log Flusher) can lead to bad write throughput if the write
> buffer is small. However, I just saw "sync" write mode is in ResultState
> when we store the query result [1]. *Is this "sync" mode really needed on
> a frame-basis*? The frame size is typically 32KB, and this could lead to
> low throughput if a user wants to pull a lot of data out of our system. If
> there is no specific need for this "sync" write, I'll submit a change to
> fix it. Thanks!
>
> [1] https://github.com/apache/asterixdb/blob/
> 77c8c79077ec33fae5944d982a337b784608aa87/hyracks-fullstack/
> hyracks/hyracks-control/hyracks-control-nc/src/main/
> java/org/apache/hyracks/control/nc/dataset/ResultState.java#L130
>
> Best regards,
> Chen Luo
>