You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Tim Armstrong (Code Review)" <ge...@cloudera.org> on 2018/06/13 21:01:32 UTC
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10709
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
......................................................................
IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
This is an attempt to either avoid the bug or make it easier to diagnose
if it reoccurs. My suspicion is that somehow boost::split() is
accessing the input string in a non-thread-safe manner, but the
implementation is opaque enough that it's not obvious how.
Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
---
M be/src/exec/hdfs-plugin-text-scanner.cc
M be/src/util/string-util-test.cc
M be/src/util/string-util.cc
M be/src/util/string-util.h
4 files changed, 38 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/10709/1
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Jun 2018 02:07:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
......................................................................
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/exec/hdfs-plugin-text-scanner.cc
File be/src/exec/hdfs-plugin-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/exec/hdfs-plugin-text-scanner.cc@107
PS1, Line 107: if (!CommaSeparatedContains(FLAGS_enabled_hdfs_text_scanner_plugins, plugin_name)) {
> Up to you, but since this isn't performance sensitive, you could use split.
I figured it would be good to avoid allocating any intermediate strings in case this doesn't fix the issue and we need to interpret more ASAN use-after-free warnings.
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc
File be/src/util/string-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@83
PS1, Line 83: // Basic tests with string present.
> Test explicitly that this doesn't whitespace-trim the entries. I don't reme
It doesn't - you have to explicitly specify the characters to split on.
I added a test at the bottom.
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@84
PS1, Line 84: EXPECT_TRUE(CommaSeparatedContains("LZO", "LZO"));
> Should we have a test with different length in string and pattern, eg. "PLU
Done
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@90
PS1, Line 90: EXPECT_FALSE(CommaSeparatedContains("", "LZO"));
> Nit: also test "," and ",,"?
Done
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:25:10 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2679/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Jun 2018 22:46:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Lars Volker, Philip Zeyliger,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10709
to look at the new patch set (#3).
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
......................................................................
IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
This is an attempt to either avoid the bug or make it easier to diagnose
if it reoccurs. My suspicion is that somehow boost::split() is
accessing the input string in a non-thread-safe manner, but the
implementation is opaque enough that it's not obvious how.
Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
---
M be/src/exec/hdfs-plugin-text-scanner.cc
M be/src/util/string-util-test.cc
M be/src/util/string-util.cc
M be/src/util/string-util.h
4 files changed, 48 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/10709/3
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Jun 2018 22:38:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Jun 2018 22:46:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
......................................................................
IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
This is an attempt to either avoid the bug or make it easier to diagnose
if it reoccurs. My suspicion is that somehow boost::split() is
accessing the input string in a non-thread-safe manner, but the
implementation is opaque enough that it's not obvious how.
Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Reviewed-on: http://gerrit.cloudera.org:8080/10709
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/hdfs-plugin-text-scanner.cc
M be/src/util/string-util-test.cc
M be/src/util/string-util.cc
M be/src/util/string-util.h
4 files changed, 48 insertions(+), 6 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/exec/hdfs-plugin-text-scanner.cc
File be/src/exec/hdfs-plugin-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/exec/hdfs-plugin-text-scanner.cc@107
PS1, Line 107: if (!CommaSeparatedContains(FLAGS_enabled_hdfs_text_scanner_plugins, plugin_name)) {
Up to you, but since this isn't performance sensitive, you could use split.h from gutil.
I think the following is similar.
vector<string> x = strings::Split(FLAGS_..., ",");
if (x.find(plugin_name) != ...::npos) { ...
}
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc
File be/src/util/string-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@83
PS1, Line 83: // Basic tests with string present.
Test explicitly that this doesn't whitespace-trim the entries. I don't remember if boost:split does or doesn't trim.
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:11:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
......................................................................
Patch Set 3: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:38:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
......................................................................
Patch Set 1:
(2 comments)
Thanks for looking into this. I noticed that we also have gutil/strings/split.h in our sources. Should we use that for a simpler implementation? I don't feel strongly about it and am ok with the custom code, too.
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc
File be/src/util/string-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@84
PS1, Line 84: EXPECT_TRUE(CommaSeparatedContains("LZO", "LZO"));
Should we have a test with different length in string and pattern, eg. "PLUGIN2", "LZO" and "A,B,C", "PLUGIN2"?
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@90
PS1, Line 90: EXPECT_FALSE(CommaSeparatedContains("", "LZO"));
Nit: also test "," and ",,"?
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:11:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
......................................................................
Patch Set 3:
Does someone want to +2?
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Jun 2018 22:36:49 +0000
Gerrit-HasComments: No