You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Hao Hao (Code Review)" <ge...@cloudera.org> on 2017/08/24 00:48:23 UTC

[kudu-CR] log block manager: use unsigned int for next block id

Hao Hao has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/7796

Change subject: log block manager: use unsigned int for next_block_id_
......................................................................

log block manager: use unsigned int for next_block_id_

next_block_id_ is used to keep track of which unique block id
should be used for block creation. Currently, it is defined as int64_t.
However, it could be updated based on the value of 'max_block_id'
which is uint64_t.

This patch changes the type of next_block_id_ to uint64_t to
avoid overflow due to conversation of unsinged int to int, which
can result in block id reused.

Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
---
M src/kudu/fs/log_block_manager.h
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/96/7796/1
-- 
To view, visit http://gerrit.cloudera.org:8080/7796
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>

[kudu-CR] log block manager: use unsigned int for next block id

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: log block manager: use unsigned int for next_block_id_
......................................................................


Patch Set 1:

(3 comments)

I've been wondering whether this deserves a standalone test. I'm inclined to say 'no' since I think the pain is only felt in specific test-only scenarios. Namely:
1) You have to be running in a gtest, so either you're using an InternalMiniCluster or a lower level unit test.
2) You have to be using the block cache singleton.
3) You have to flush some data.
4) You have to restart the block manager some number of times.

BTW, KUDU-1538 is related and perhaps should be referenced in the commit description.

http://gerrit.cloudera.org:8080/#/c/7796/1//COMMIT_MSG
Commit Message:

Line 14: This patch changes the type of next_block_id_ to uint64_t to
Should probably explain why we're switching it to uint64_t instead of switching every other site to int64_t (since the convention in Kudu is to use int64_t for large integer values).

The reason is that block IDs are defined as uint64_t both on disk (fs.proto) and in memory (block_id.h), so it makes more sense to treat them as such in the LBM than to convert them properly.


PS1, Line 15: conversation
conversion


PS1, Line 16: in block id reused.
in the reuse of an existing block ID.


-- 
To view, visit http://gerrit.cloudera.org:8080/7796
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes

[kudu-CR] log block manager: use unsigned int for next block id

Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/7796

to look at the new patch set (#2).

Change subject: log block manager: use unsigned int for next_block_id_
......................................................................

log block manager: use unsigned int for next_block_id_

KUDU-1538 introduced 'next_block_id_' to keep track of unique block
ID that should be used for block creation. Currently, it is defined
as int64_t. However, it could be updated based on the value of
'max_block_id' which is uint64_t. Moreover, block IDs are defined
as uint64_t both on disk (fs.proto) and in memory (block_id.h), so
it makes more sense to treat 'next_block_id_' as uint64_t than to
convert it properly.

This patch changes the type of 'next_block_id_' to uint64_t to
avoid overflow due to conversion of unsinged int to int, which
can result in the reuse of an existing block ID. It does not add
a standalone test case because the failure can most likely occur
in specific test-only scenarios.

Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
---
M src/kudu/fs/log_block_manager.h
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/96/7796/2
-- 
To view, visit http://gerrit.cloudera.org:8080/7796
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins

[kudu-CR] log block manager: use unsigned int for next block id

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has submitted this change and it was merged.

Change subject: log block manager: use unsigned int for next_block_id_
......................................................................


log block manager: use unsigned int for next_block_id_

KUDU-1538 introduced 'next_block_id_' to keep track of unique block
ID that should be used for block creation. Currently, it is defined
as int64_t. However, it could be updated based on the value of
'max_block_id' which is uint64_t. Since block IDs are defined as
uint64_t both on disk (fs.proto) and in memory (block_id.h), it
makes more sense to treat 'next_block_id_' as uint64_t rather than
to convert it correctly to int64_t everywhere.

This patch changes the type of 'next_block_id_' to uint64_t to
avoid overflow due to conversion of unsigned int to int, which
can result in the reuse of an existing block ID. It does not add
a standalone test case because the failure is most likely to occur
in specific test-only scenarios.

Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Reviewed-on: http://gerrit.cloudera.org:8080/7796
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Reviewed-by: Todd Lipcon <to...@apache.org>
Tested-by: Kudu Jenkins
---
M src/kudu/fs/file_block_manager.h
M src/kudu/fs/log_block_manager.h
2 files changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Adar Dembo: Looks good to me, approved
  Todd Lipcon: Looks good to me, approved
  Kudu Jenkins: Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/7796
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Gerrit-PatchSet: 4
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] log block manager: use unsigned int for next block id

Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change.

Change subject: log block manager: use unsigned int for next_block_id_
......................................................................


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/7796/1//COMMIT_MSG
Commit Message:

Line 14: it makes more sense to treat 'next_block_id_' as uint64_t than to
> Should probably explain why we're switching it to uint64_t instead of switc
Done


PS1, Line 15: 
> conversion
Done


PS1, Line 16: 
> in the reuse of an existing block ID.
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/7796
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes

[kudu-CR] log block manager: use unsigned int for next block id

Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change.

Change subject: log block manager: use unsigned int for next_block_id_
......................................................................


Patch Set 3:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/7796/2//COMMIT_MSG
Commit Message:

PS2, Line 12: Since block IDs are
> "Since block IDs..."
Done


PS2, Line 13: t
> drop this
Done


PS2, Line 14: her 
> rather than
Done


PS2, Line 15: it corre
> correctly to int64_t everywhere
Done


PS2, Line 18: unsigned 
> typo, should be unsigned
Done


PS2, Line 20: is 
> is
Done


PS2, Line 20: o occ
> to occur
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/7796
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes

[kudu-CR] log block manager: use unsigned int for next block id

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: log block manager: use unsigned int for next_block_id_
......................................................................


Patch Set 2:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/7796/2//COMMIT_MSG
Commit Message:

PS2, Line 12: Moreover, block IDs
"Since block IDs..."


PS2, Line 13: , so
drop this


PS2, Line 14: than
rather than


PS2, Line 15: properly
correctly to int64_t everywhere


PS2, Line 20: can
is


PS2, Line 20: occur
to occur


-- 
To view, visit http://gerrit.cloudera.org:8080/7796
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes

[kudu-CR] log block manager: use unsigned int for next block id

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.

Change subject: log block manager: use unsigned int for next_block_id_
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/7796/2//COMMIT_MSG
Commit Message:

PS2, Line 18: unsinged 
typo, should be unsigned


http://gerrit.cloudera.org:8080/#/c/7796/2/src/kudu/fs/log_block_manager.h
File src/kudu/fs/log_block_manager.h:

PS2, Line 411: AtomicInt
maybe, to be on the safe side, use std::atomic<uint64_t> here to have well-defined behavior in case of overflow?


-- 
To view, visit http://gerrit.cloudera.org:8080/7796
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes

[kudu-CR] log block manager: use unsigned int for next block id

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: log block manager: use unsigned int for next_block_id_
......................................................................


Patch Set 3: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/7796
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] log block manager: use unsigned int for next block id

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: log block manager: use unsigned int for next_block_id_
......................................................................


Patch Set 3: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/7796
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: No

[kudu-CR] log block manager: use unsigned int for next block id

Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/7796

to look at the new patch set (#3).

Change subject: log block manager: use unsigned int for next_block_id_
......................................................................

log block manager: use unsigned int for next_block_id_

KUDU-1538 introduced 'next_block_id_' to keep track of unique block
ID that should be used for block creation. Currently, it is defined
as int64_t. However, it could be updated based on the value of
'max_block_id' which is uint64_t. Since block IDs are defined as
uint64_t both on disk (fs.proto) and in memory (block_id.h), it
makes more sense to treat 'next_block_id_' as uint64_t rather than
to convert it correctly to int64_t everywhere.

This patch changes the type of 'next_block_id_' to uint64_t to
avoid overflow due to conversion of unsigned int to int, which
can result in the reuse of an existing block ID. It does not add
a standalone test case because the failure is most likely to occur
in specific test-only scenarios.

Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
---
M src/kudu/fs/file_block_manager.h
M src/kudu/fs/log_block_manager.h
2 files changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/96/7796/3
-- 
To view, visit http://gerrit.cloudera.org:8080/7796
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins

[kudu-CR] log block manager: use unsigned int for next block id

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.

Change subject: log block manager: use unsigned int for next_block_id_
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/7796/2/src/kudu/fs/log_block_manager.h
File src/kudu/fs/log_block_manager.h:

PS2, Line 411: AtomicInt
> I'd prefer if we kept the scope of this patch limited, and we already use u
Sure, that makes sense.


-- 
To view, visit http://gerrit.cloudera.org:8080/7796
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes

[kudu-CR] log block manager: use unsigned int for next block id

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: log block manager: use unsigned int for next_block_id_
......................................................................


Patch Set 2:

(1 comment)

The test failures are unrelated; some test machine clocks are unsynchronized for some reason.

http://gerrit.cloudera.org:8080/#/c/7796/2/src/kudu/fs/log_block_manager.h
File src/kudu/fs/log_block_manager.h:

PS2, Line 411: AtomicInt
> maybe, to be on the safe side, use std::atomic<uint64_t> here to have well-
I'd prefer if we kept the scope of this patch limited, and we already use util/atomic throughout the block manager code.


-- 
To view, visit http://gerrit.cloudera.org:8080/7796
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib315b20719ef529331304df5c56c4242902524d4
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes