You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Ivan Kelly (JIRA)" <ji...@apache.org> on 2011/07/14 10:31:59 UTC
[jira] [Created] (BOOKKEEPER-33) Add length and offset parameter to
addEntry
Add length and offset parameter to addEntry
-------------------------------------------
Key: BOOKKEEPER-33
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-33
Project: Bookkeeper
Issue Type: Improvement
Reporter: Ivan Kelly
<from email to dev list>
I'm having an issue with the LedgerHandle#addEntry api.
[1] best illustrates it. I'm buffering namenode transactions in the stream and only transmitting when either flush is called or I have enough data to pass my threshold. This means I have a byte buffer in my class which I fill up as new transactions come in. When I transmit, I set this buffer as an entry to bookkeeper. I.e. N whole namenode transactions will be contained in 1 single bk entry.
The problem is this byte buffer (DataOutputBuffer in this case). I reuse the same buffer over and over. But this buffer has a fixed size. If I transmit before it is full, the whole buffer size will be transmitted anyhow. If the buffer is being reused, this will retransmit old transactions out of order. For example, in the first use, the buffer fills with, [a,b,c,d,e] and adds this as an entry and resets the byte buffer. Then transaction f is added and flushed, in this case [f,b,c,d,e] is not transmitted.
What I need is the ability to set offset and length in the byte[] passed to addEntry. Is there a reason this wasn't added in the initial implementation? If not, and if you agree this is a valid usecase, ill open a JIRA and add this functionality. Im getting around this now by doing an extra Array.copyOf which is less than ideal.
</from email>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-33) Add length and offset parameter
to addEntry
Posted by "Flavio Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-33?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081299#comment-13081299 ]
Flavio Junqueira commented on BOOKKEEPER-33:
--------------------------------------------
My +1 was precipitated, BookieReadWriteTest hangs for me, but interestingly it passes if I turn console logging on.
> Add length and offset parameter to addEntry
> -------------------------------------------
>
> Key: BOOKKEEPER-33
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-33
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Attachments: BK-33.diff
>
>
> <from email to dev list>
> I'm having an issue with the LedgerHandle#addEntry api.
> [1] best illustrates it. I'm buffering namenode transactions in the stream and only transmitting when either flush is called or I have enough data to pass my threshold. This means I have a byte buffer in my class which I fill up as new transactions come in. When I transmit, I set this buffer as an entry to bookkeeper. I.e. N whole namenode transactions will be contained in 1 single bk entry.
> The problem is this byte buffer (DataOutputBuffer in this case). I reuse the same buffer over and over. But this buffer has a fixed size. If I transmit before it is full, the whole buffer size will be transmitted anyhow. If the buffer is being reused, this will retransmit old transactions out of order. For example, in the first use, the buffer fills with, [a,b,c,d,e] and adds this as an entry and resets the byte buffer. Then transaction f is added and flushed, in this case [f,b,c,d,e] is not transmitted.
> What I need is the ability to set offset and length in the byte[] passed to addEntry. Is there a reason this wasn't added in the initial implementation? If not, and if you agree this is a valid usecase, ill open a JIRA and add this functionality. Im getting around this now by doing an extra Array.copyOf which is less than ideal.
> </from email>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (BOOKKEEPER-33) Add length and offset
parameter to addEntry
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-33?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065114#comment-13065114 ]
Ivan Kelly edited comment on BOOKKEEPER-33 at 7/14/11 8:41 AM:
---------------------------------------------------------------
{quote}
// from Utkarsh on dev list
Is it worth using ByteBuffer instead of passing around offset and limit
separately?
{quote}
I wouldn't like to have
{code}
public long addEntry(byte[] data) throws InterruptedException, BKException {
{code}
and
{code}
public long addEntry(ByteBuffer data) throws InterruptedException, BKException {
{code}
This looks a little imbalanced to me. I'd prefer to either have only the ByteBuffer call, or to have:
{code}
public long addEntry(byte[] data) throws InterruptedException, BKException {
public long addEntry(byte[] data, int offset, int length) throws InterruptedException, BKException {
{code}
Internally we could use bytebuffers easily though. If we simply add a ByteBuffer method to the api, we should change all the other methods to work with byte buffers too.
was (Author: ikelly):
{quote}
// from Utkarsh on dev list
Is it worth using ByteBuffer instead of passing around offset and limit
separately?
{quote}
I wouldn't like to have
{code}
public long addEntry(byte[] data) throws InterruptedException, BKException {
{code}
and
{code}
public long addEntry(ByteBuffer data) throws InterruptedException, BKException {
{code}
This looks a little imbalanced to me. I'd prefer to either have only the ByteBuffer call, or to have:
{code}
public long addEntry(byte[] data) throws InterruptedException, BKException {
public long addEntry(byte[] data, int offset, int length) throws InterruptedException, BKException {
{code}
Internally we could use bytebuffers easily though.
> Add length and offset parameter to addEntry
> -------------------------------------------
>
> Key: BOOKKEEPER-33
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-33
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
>
> <from email to dev list>
> I'm having an issue with the LedgerHandle#addEntry api.
> [1] best illustrates it. I'm buffering namenode transactions in the stream and only transmitting when either flush is called or I have enough data to pass my threshold. This means I have a byte buffer in my class which I fill up as new transactions come in. When I transmit, I set this buffer as an entry to bookkeeper. I.e. N whole namenode transactions will be contained in 1 single bk entry.
> The problem is this byte buffer (DataOutputBuffer in this case). I reuse the same buffer over and over. But this buffer has a fixed size. If I transmit before it is full, the whole buffer size will be transmitted anyhow. If the buffer is being reused, this will retransmit old transactions out of order. For example, in the first use, the buffer fills with, [a,b,c,d,e] and adds this as an entry and resets the byte buffer. Then transaction f is added and flushed, in this case [f,b,c,d,e] is not transmitted.
> What I need is the ability to set offset and length in the byte[] passed to addEntry. Is there a reason this wasn't added in the initial implementation? If not, and if you agree this is a valid usecase, ill open a JIRA and add this functionality. Im getting around this now by doing an extra Array.copyOf which is less than ideal.
> </from email>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (BOOKKEEPER-33) Add length and offset
parameter to addEntry
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-33?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065114#comment-13065114 ]
Ivan Kelly edited comment on BOOKKEEPER-33 at 7/14/11 8:40 AM:
---------------------------------------------------------------
{quote}
// from Utkarsh on dev list
Is it worth using ByteBuffer instead of passing around offset and limit
separately?
{quote}
I wouldn't like to have
{code}
public long addEntry(byte[] data) throws InterruptedException, BKException {
{code}
and
{code}
public long addEntry(ByteBuffer data) throws InterruptedException, BKException {
{code}
This looks a little imbalanced to me. I'd prefer to either have only the ByteBuffer call, or to have:
{code}
public long addEntry(byte[] data) throws InterruptedException, BKException {
public long addEntry(byte[] data, int offset, int length) throws InterruptedException, BKException {
{code}
Internally we could use bytebuffers easily though.
was (Author: ikelly):
{quote}
// from Utkarsh on dev list
Is it worth using ByteBuffer instead of passing around offset and limit
separately?
{quote}
I wouldn't like to have
{code}
public long addEntry(byte[] data) throws InterruptedException, BKException {
{code}
and
{code}
public long addEntry(ByteBuffer data) throws InterruptedException, BKException {
{code}
This looks a little imbalanced to me. I'd prefer to either have only the ByteBuffer call, or to have:
{code}
public long addEntry(byte[] data) throws InterruptedException, BKException {
public long addEntry(byte[] data, int offset, int length) throws InterruptedException, BKException {
{code}
Internally we could use bytebuffers easily though.
> Add length and offset parameter to addEntry
> -------------------------------------------
>
> Key: BOOKKEEPER-33
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-33
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
>
> <from email to dev list>
> I'm having an issue with the LedgerHandle#addEntry api.
> [1] best illustrates it. I'm buffering namenode transactions in the stream and only transmitting when either flush is called or I have enough data to pass my threshold. This means I have a byte buffer in my class which I fill up as new transactions come in. When I transmit, I set this buffer as an entry to bookkeeper. I.e. N whole namenode transactions will be contained in 1 single bk entry.
> The problem is this byte buffer (DataOutputBuffer in this case). I reuse the same buffer over and over. But this buffer has a fixed size. If I transmit before it is full, the whole buffer size will be transmitted anyhow. If the buffer is being reused, this will retransmit old transactions out of order. For example, in the first use, the buffer fills with, [a,b,c,d,e] and adds this as an entry and resets the byte buffer. Then transaction f is added and flushed, in this case [f,b,c,d,e] is not transmitted.
> What I need is the ability to set offset and length in the byte[] passed to addEntry. Is there a reason this wasn't added in the initial implementation? If not, and if you agree this is a valid usecase, ill open a JIRA and add this functionality. Im getting around this now by doing an extra Array.copyOf which is less than ideal.
> </from email>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-33) Add length and offset parameter
to addEntry
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-33?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065114#comment-13065114 ]
Ivan Kelly commented on BOOKKEEPER-33:
--------------------------------------
{quote}
// from Utkarsh on dev list
Is it worth using ByteBuffer instead of passing around offset and limit
separately?
{quote}
I wouldn't like to have
{code}
public long addEntry(byte[] data) throws InterruptedException, BKException {
{code}
and
{code}
public long addEntry(ByteBuffer data) throws InterruptedException, BKException {
{code}
This looks a little imbalanced to me. I'd prefer to either have only the ByteBuffer call, or to have:
{code}
public long addEntry(byte[] data) throws InterruptedException, BKException {
public long addEntry(byte[] data, int offset, int length) throws InterruptedException, BKException {
{code}
Internally we could use bytebuffers easily though.
> Add length and offset parameter to addEntry
> -------------------------------------------
>
> Key: BOOKKEEPER-33
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-33
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
>
> <from email to dev list>
> I'm having an issue with the LedgerHandle#addEntry api.
> [1] best illustrates it. I'm buffering namenode transactions in the stream and only transmitting when either flush is called or I have enough data to pass my threshold. This means I have a byte buffer in my class which I fill up as new transactions come in. When I transmit, I set this buffer as an entry to bookkeeper. I.e. N whole namenode transactions will be contained in 1 single bk entry.
> The problem is this byte buffer (DataOutputBuffer in this case). I reuse the same buffer over and over. But this buffer has a fixed size. If I transmit before it is full, the whole buffer size will be transmitted anyhow. If the buffer is being reused, this will retransmit old transactions out of order. For example, in the first use, the buffer fills with, [a,b,c,d,e] and adds this as an entry and resets the byte buffer. Then transaction f is added and flushed, in this case [f,b,c,d,e] is not transmitted.
> What I need is the ability to set offset and length in the byte[] passed to addEntry. Is there a reason this wasn't added in the initial implementation? If not, and if you agree this is a valid usecase, ill open a JIRA and add this functionality. Im getting around this now by doing an extra Array.copyOf which is less than ideal.
> </from email>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-33) Add length and offset parameter
to addEntry
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-33?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082453#comment-13082453 ]
Ivan Kelly commented on BOOKKEEPER-33:
--------------------------------------
That looks very like the netty problem we used to have. I'll kick it off in a loop overnight to see if I can hit the same thing. Have you definitely applied on a clean up to date checkout of trunk?
> Add length and offset parameter to addEntry
> -------------------------------------------
>
> Key: BOOKKEEPER-33
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-33
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Fix For: 3.4.0
>
> Attachments: BK-33.diff, bookkeeper-33-trace.txt
>
>
> <from email to dev list>
> I'm having an issue with the LedgerHandle#addEntry api.
> [1] best illustrates it. I'm buffering namenode transactions in the stream and only transmitting when either flush is called or I have enough data to pass my threshold. This means I have a byte buffer in my class which I fill up as new transactions come in. When I transmit, I set this buffer as an entry to bookkeeper. I.e. N whole namenode transactions will be contained in 1 single bk entry.
> The problem is this byte buffer (DataOutputBuffer in this case). I reuse the same buffer over and over. But this buffer has a fixed size. If I transmit before it is full, the whole buffer size will be transmitted anyhow. If the buffer is being reused, this will retransmit old transactions out of order. For example, in the first use, the buffer fills with, [a,b,c,d,e] and adds this as an entry and resets the byte buffer. Then transaction f is added and flushed, in this case [f,b,c,d,e] is not transmitted.
> What I need is the ability to set offset and length in the byte[] passed to addEntry. Is there a reason this wasn't added in the initial implementation? If not, and if you agree this is a valid usecase, ill open a JIRA and add this functionality. Im getting around this now by doing an extra Array.copyOf which is less than ideal.
> </from email>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-33) Add length and offset parameter to
addEntry
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-33?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated BOOKKEEPER-33:
---------------------------------
Attachment: BK-33.diff
Ready for review
> Add length and offset parameter to addEntry
> -------------------------------------------
>
> Key: BOOKKEEPER-33
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-33
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Attachments: BK-33.diff
>
>
> <from email to dev list>
> I'm having an issue with the LedgerHandle#addEntry api.
> [1] best illustrates it. I'm buffering namenode transactions in the stream and only transmitting when either flush is called or I have enough data to pass my threshold. This means I have a byte buffer in my class which I fill up as new transactions come in. When I transmit, I set this buffer as an entry to bookkeeper. I.e. N whole namenode transactions will be contained in 1 single bk entry.
> The problem is this byte buffer (DataOutputBuffer in this case). I reuse the same buffer over and over. But this buffer has a fixed size. If I transmit before it is full, the whole buffer size will be transmitted anyhow. If the buffer is being reused, this will retransmit old transactions out of order. For example, in the first use, the buffer fills with, [a,b,c,d,e] and adds this as an entry and resets the byte buffer. Then transaction f is added and flushed, in this case [f,b,c,d,e] is not transmitted.
> What I need is the ability to set offset and length in the byte[] passed to addEntry. Is there a reason this wasn't added in the initial implementation? If not, and if you agree this is a valid usecase, ill open a JIRA and add this functionality. Im getting around this now by doing an extra Array.copyOf which is less than ideal.
> </from email>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-33) Add length and offset parameter to
addEntry
Posted by "Flavio Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-33?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flavio Junqueira updated BOOKKEEPER-33:
---------------------------------------
Attachment: bookkeeper-33-trace.txt
Stack trace of a test run that halted.
> Add length and offset parameter to addEntry
> -------------------------------------------
>
> Key: BOOKKEEPER-33
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-33
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Attachments: BK-33.diff, bookkeeper-33-trace.txt
>
>
> <from email to dev list>
> I'm having an issue with the LedgerHandle#addEntry api.
> [1] best illustrates it. I'm buffering namenode transactions in the stream and only transmitting when either flush is called or I have enough data to pass my threshold. This means I have a byte buffer in my class which I fill up as new transactions come in. When I transmit, I set this buffer as an entry to bookkeeper. I.e. N whole namenode transactions will be contained in 1 single bk entry.
> The problem is this byte buffer (DataOutputBuffer in this case). I reuse the same buffer over and over. But this buffer has a fixed size. If I transmit before it is full, the whole buffer size will be transmitted anyhow. If the buffer is being reused, this will retransmit old transactions out of order. For example, in the first use, the buffer fills with, [a,b,c,d,e] and adds this as an entry and resets the byte buffer. Then transaction f is added and flushed, in this case [f,b,c,d,e] is not transmitted.
> What I need is the ability to set offset and length in the byte[] passed to addEntry. Is there a reason this wasn't added in the initial implementation? If not, and if you agree this is a valid usecase, ill open a JIRA and add this functionality. Im getting around this now by doing an extra Array.copyOf which is less than ideal.
> </from email>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-33) Add length and offset parameter
to addEntry
Posted by "Flavio Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-33?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081094#comment-13081094 ]
Flavio Junqueira commented on BOOKKEEPER-33:
--------------------------------------------
+1, looks good!
> Add length and offset parameter to addEntry
> -------------------------------------------
>
> Key: BOOKKEEPER-33
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-33
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Attachments: BK-33.diff
>
>
> <from email to dev list>
> I'm having an issue with the LedgerHandle#addEntry api.
> [1] best illustrates it. I'm buffering namenode transactions in the stream and only transmitting when either flush is called or I have enough data to pass my threshold. This means I have a byte buffer in my class which I fill up as new transactions come in. When I transmit, I set this buffer as an entry to bookkeeper. I.e. N whole namenode transactions will be contained in 1 single bk entry.
> The problem is this byte buffer (DataOutputBuffer in this case). I reuse the same buffer over and over. But this buffer has a fixed size. If I transmit before it is full, the whole buffer size will be transmitted anyhow. If the buffer is being reused, this will retransmit old transactions out of order. For example, in the first use, the buffer fills with, [a,b,c,d,e] and adds this as an entry and resets the byte buffer. Then transaction f is added and flushed, in this case [f,b,c,d,e] is not transmitted.
> What I need is the ability to set offset and length in the byte[] passed to addEntry. Is there a reason this wasn't added in the initial implementation? If not, and if you agree this is a valid usecase, ill open a JIRA and add this functionality. Im getting around this now by doing an extra Array.copyOf which is less than ideal.
> </from email>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-33) Add length and offset parameter
to addEntry
Posted by "Utkarsh Srivastava (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-33?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065400#comment-13065400 ]
Utkarsh Srivastava commented on BOOKKEEPER-33:
----------------------------------------------
Seems like adding the offset and length is a good enhancement in itself.
There is another unrelated enhancement which is to take ByteBuffer directly in case of NIO clients that read it from somewhere and want to log it to Bookkeeper without copying. Agreed that a second enhancement is probably a separate JIRA.
> Add length and offset parameter to addEntry
> -------------------------------------------
>
> Key: BOOKKEEPER-33
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-33
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
>
> <from email to dev list>
> I'm having an issue with the LedgerHandle#addEntry api.
> [1] best illustrates it. I'm buffering namenode transactions in the stream and only transmitting when either flush is called or I have enough data to pass my threshold. This means I have a byte buffer in my class which I fill up as new transactions come in. When I transmit, I set this buffer as an entry to bookkeeper. I.e. N whole namenode transactions will be contained in 1 single bk entry.
> The problem is this byte buffer (DataOutputBuffer in this case). I reuse the same buffer over and over. But this buffer has a fixed size. If I transmit before it is full, the whole buffer size will be transmitted anyhow. If the buffer is being reused, this will retransmit old transactions out of order. For example, in the first use, the buffer fills with, [a,b,c,d,e] and adds this as an entry and resets the byte buffer. Then transaction f is added and flushed, in this case [f,b,c,d,e] is not transmitted.
> What I need is the ability to set offset and length in the byte[] passed to addEntry. Is there a reason this wasn't added in the initial implementation? If not, and if you agree this is a valid usecase, ill open a JIRA and add this functionality. Im getting around this now by doing an extra Array.copyOf which is less than ideal.
> </from email>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-33) Add length and offset parameter
to addEntry
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-33?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082276#comment-13082276 ]
Ivan Kelly commented on BOOKKEEPER-33:
--------------------------------------
Where does it hang? (kill -QUIT <pid> will give a stacktrace) Have you tried it on a different machine? I've run it in a loop with logging off and it hasn't hung for me.
One thing I have noticed in these tests is that Object#wait() & Object#notify() are being used. This can be problematic if notify() is called on the callback thread before wait() is called on the client thread. It'll hang forever. These should be changed to CountdownLatch in all tests, but thats another JIRA.
> Add length and offset parameter to addEntry
> -------------------------------------------
>
> Key: BOOKKEEPER-33
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-33
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Attachments: BK-33.diff
>
>
> <from email to dev list>
> I'm having an issue with the LedgerHandle#addEntry api.
> [1] best illustrates it. I'm buffering namenode transactions in the stream and only transmitting when either flush is called or I have enough data to pass my threshold. This means I have a byte buffer in my class which I fill up as new transactions come in. When I transmit, I set this buffer as an entry to bookkeeper. I.e. N whole namenode transactions will be contained in 1 single bk entry.
> The problem is this byte buffer (DataOutputBuffer in this case). I reuse the same buffer over and over. But this buffer has a fixed size. If I transmit before it is full, the whole buffer size will be transmitted anyhow. If the buffer is being reused, this will retransmit old transactions out of order. For example, in the first use, the buffer fills with, [a,b,c,d,e] and adds this as an entry and resets the byte buffer. Then transaction f is added and flushed, in this case [f,b,c,d,e] is not transmitted.
> What I need is the ability to set offset and length in the byte[] passed to addEntry. Is there a reason this wasn't added in the initial implementation? If not, and if you agree this is a valid usecase, ill open a JIRA and add this functionality. Im getting around this now by doing an extra Array.copyOf which is less than ideal.
> </from email>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-33) Add length and offset parameter to
addEntry
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-33?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated BOOKKEEPER-33:
---------------------------------
Fix Version/s: 3.4.0
> Add length and offset parameter to addEntry
> -------------------------------------------
>
> Key: BOOKKEEPER-33
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-33
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Fix For: 3.4.0
>
> Attachments: BK-33.diff, bookkeeper-33-trace.txt
>
>
> <from email to dev list>
> I'm having an issue with the LedgerHandle#addEntry api.
> [1] best illustrates it. I'm buffering namenode transactions in the stream and only transmitting when either flush is called or I have enough data to pass my threshold. This means I have a byte buffer in my class which I fill up as new transactions come in. When I transmit, I set this buffer as an entry to bookkeeper. I.e. N whole namenode transactions will be contained in 1 single bk entry.
> The problem is this byte buffer (DataOutputBuffer in this case). I reuse the same buffer over and over. But this buffer has a fixed size. If I transmit before it is full, the whole buffer size will be transmitted anyhow. If the buffer is being reused, this will retransmit old transactions out of order. For example, in the first use, the buffer fills with, [a,b,c,d,e] and adds this as an entry and resets the byte buffer. Then transaction f is added and flushed, in this case [f,b,c,d,e] is not transmitted.
> What I need is the ability to set offset and length in the byte[] passed to addEntry. Is there a reason this wasn't added in the initial implementation? If not, and if you agree this is a valid usecase, ill open a JIRA and add this functionality. Im getting around this now by doing an extra Array.copyOf which is less than ideal.
> </from email>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira