You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Hao Hao (Code Review)" <ge...@cloudera.org> on 2019/10/01 05:14:38 UTC
[kudu-CR] [java] process communicates via protobuf-based protocol
Hao Hao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14329
Change subject: [java] process communicates via protobuf-based protocol
......................................................................
[java] process communicates via protobuf-based protocol
This commit adds a java tool that can communicate with other process
via protobuf-based protocol over standard input/output. It is useful
in cases a Kudu process (e.g. master) needs to talk to third-party
libraries written in Java.
This tool has: 1) a single reader(producer) thread that continuously
reads protobuf-based messages (with optional JSON encoding) from the
standard input and puts the messages to a FIFO blocking queue;
2) multiple writer(consumer) threads that continuously retrieve the
messages from the queue, process them and write the responses to the
standard output.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface to specify how to handle the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
M CMakeLists.txt
M java/gradle/dependencies.gradle
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLine.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessRuntimeException.java
A java/kudu-subprocess/src/main/resources/log4j.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProducerConsumer.java
M java/settings.gradle
A src/kudu/subprocess/CMakeLists.txt
A src/kudu/subprocess/subprocess.proto
20 files changed, 1,481 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/1
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 1
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 6:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/build.gradle
File java/kudu-subprocess/build.gradle:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/build.gradle@25
PS6, Line 25: compile libs.hadoopCommon
> Is this included just for Configuration.Java? There should be something mor
Hmm, it seems I cannot find another more lightweight to include than hadoopCommon. Which one you are talking about
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/build.gradle@56
PS6, Line 56: configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }
> The shadow plugin, included above, should handle the shading of this jar fo
Done
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java@45
PS6, Line 45: BasicSubprocessor
> nit: there's not other subprocessor type, so maybe just call this Subproces
Done
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java@96
PS6, Line 96: catch (IOException e) {
: LOG.error("Unable to close the underlying output stream", e);
: }
> Will this ever catch anything if we're doing System.exit(1) when we hit an
Are you asking if close() will be called with System.exit(1)? The oracle Java doc said "The try-with-resources statement ensures that each resource is closed at the end of the statement". However, the statement is not very clear about System.exit() situation. So I did an experiment to extend BufferedInputStream with customized close() to verify if close() is called and if the catch block get executed when close() throw IOException. The answer is 'Yes' for both.
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@44
PS6, Line 44:
: MessageProcessor(long maxMessageBytes) {
: this.maxMessageBytes = maxMessageBytes;
: }
:
: MessageProcessor(long maxMessageBytes,
: BufferedInputStream in) {
: Preconditions.checkNotNull(in);
: this.maxMessageBytes = maxMessageBytes;
: this.in = in;
: }
:
: MessageProcessor(long maxMessageBytes,
: BufferedOutputStream out) {
: Preconditions.checkNotNull(out);
: this.maxMessageBytes = maxMessageBytes;
: this.out = out;
: }
> nit: Why do we need so many constructors if we have MessageProcessor(long m
Done
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@52
PS6, Line 52: this.maxMessageBytes = maxMessageBytes;
: this.in = in;
: }
:
: MessageProcessor(long maxMessageBytes,
: BufferedOutputStream out) {
: Preconditions.checkNotNull(out);
: this.maxMessageBytes = maxMessageBytes;
: this.out = out;
: }
:
: MessageProcessor(long maxMessageBytes,
: BufferedInputStream in,
: BufferedOutputStream out) {
: Preconditions.checkNotNull(in);
: Preconditions.checkNotNull(out);
: this.maxMessageBytes = maxMessageBytes;
: this.in = in;
: this.out = out;
> nit: these could use setInputStream() and setOutputStream()
Done
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@92
PS6, Line 92: * @throws IllegalStateException if the message is malformed
> No longer true?
This is refer to L106 and L113
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@98
PS6, Line 98: interrupted = true;
> If we are interrupted, but we try and eventually succeed at putting the mes
Are you asking should we do that? I don't quite understand why we still want to break out the larger loop in this case? Why not proceed to the next message?
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@104
PS6, Line 104: Restore
> nit: "Return"? "Propagate"?
Done
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@41
PS6, Line 41: IllegalArgumentException
> InvalidProtocolBufferException?
Done
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@69
PS6, Line 69: String confFilePath = cmd.getOptionValue(CONF_LONG_OPTION);
> Requiring a configuration file means that Kudu deployments that want to use
I feel configuration file is more flexible if we have more (or less) configs in future. And we can programmatically generate the config file as we do for sentry and hms (Similar for Ranger configurations). But I can update that if you think strongly otherwise.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 6
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 17 Dec 2019 03:19:20 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 18:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/build.gradle
File java/kudu-subprocess/build.gradle:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/build.gradle@46
PS17, Line 46: jar {
: manifest {
: attributes(
: 'Main-Class': 'org.apache.kudu.subprocess.EchoSubprocessMain'
: )
: }
: }
> Yeah, EchoSubprocessMain is the default protocol supported for kudu-subproc
Hrm, should we ship all the subprocess work as a library, and instead move the EchoSubprocessMain stuff into its own echo-subprocess directory? Then we wouldn't have to build an echo processor whenever we want to build another Kudu subprocess JAR, right?
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@33
PS17, Line 33: private int queueSize;
> Done
I meant, why bother defining constants for "q", "w", "m", and all the descriptions if they're only used once?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 18
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 06 Feb 2020 23:17:38 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#5).
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2791: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate with other process
via protobuf-based protocol over standard input/output. It is useful
in cases a Kudu process (e.g. master) needs to talk to third-party
libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOExcpetion is fatal if encountered when reading/writing to the pipe
and can cause the tool to exit, while an error message is responded
for malformed protobuf message.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and update the SubprocessConfiguration to specify how to handle
the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLine.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderConsumer.java
M java/settings.gradle
16 files changed, 1,365 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/5
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 5
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 2:
(2 comments)
Just passing through, will wait for an update to actually review.
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG@15
PS1, Line 15: optional JSON encoding) from
> I agree I don't see a strong need for it. We could remove all the configura
Agreed with Andrew and Grant. The "subprocess protocol" module must support both JSON and PB, but that doesn't mean this new code needs to support both. We can get good debuggability using the usual methods (i.e. VLOG(1) << SecureDebugString(...)).
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@55
PS1, Line 55: static void run(SubprocessConfiguration conf) {
Could be private? Or if this visibility is only for testing, annotate with @InterfaceAudience.LimitedPrivate("Test")
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 23 Oct 2019 04:28:25 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 15:
(17 comments)
> Patch Set 14:
>
> (17 comments)
>
> Overall looks much cleaner and simpler than the last revision I looked at. Thanks for making all of the changes.
>
> Note: I didn't review the test code.
Thanks a lot for the review!
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolHandler.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolHandler.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolHandler.java@34
PS14, Line 34: respBuilde
> Nit: elsewhere in the C++ code we generally use 'resp' as the prefix for th
Done
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@34
PS14, Line 34: * Util class for reading and writing protobuf messages.
> Nit: reading and writing protobuf messages.
Done
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@98
PS14, Line 98: expect
> expected
Done
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@105
PS14, Line 105: will b
> will
Done
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@79
PS14, Line 79: exit
> Nit: exiting
Done
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@90
PS14, Line 90: if (data.length == 0) {
> Under what circumstances would we expect to see an empty message? Doc?
We don't expect to see empty messages in general. Updated the comment.
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@37
PS14, Line 37: o
> one
Done
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@65
PS14, Line 65: while (true) {
> Why do we have to test this condition? Why not just while (true)? If we're
Hmm, you're right, missed this one, thanks!
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@92
PS14, Line 92: * Constructs a message with the given error status.
> Nit: resp
Done
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@123
PS14, Line 123: new String(data, StandardCharsets.UTF_8)), e);
> This method is simple enough that I think you can inline it into run().
Done
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@91
PS14, Line 91:
:
:
:
> What do you think of passing the entire CommandLine into the SubprocessConf
If I understand it correctly, you are suggesting to combine CommandLine with SubprocessConfiguration, so that the caller only need to care about SubprocessConfiguration? If so, looks good to me, updated accordingly.
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@42
PS14, Line 42: WRITER_
> Nit: add a space before
Done
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@44
PS14, Line 44: "Maximum number of threads in the writer thread pool for subprocess";
> Nit: maybe you can shorten the Javadoc for the below getters:
Done
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@52
PS14, Line 52: // Exit the program with a nonzero status code if unexpected exception(s)
> This isn't actually "visible" for testing. The accessor is (and you've anno
Done
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@56
PS14, Line 56: };
: private boolean injectInterrupt = false;
:
: SubprocessExecutor() {
: }
:
> Can this be defined in the declaration of errorHandler itself? Could it be
Done
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@108
PS14, Line 108: for (int i = 0; i < maxWriterThread; i++) {
> Can you put the readerFuture into this array too? Then you can simplify the
The problem is certain tests depends on the failure of the reader task. If adding readerFuture along with writerFutures, we will get TimeoutException instead of ExecutionException as we are waiting for all tasks to finish.
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java@44
PS14, Line 44: @Override
> Do we need to provide overrides for the other two write() variants, or for
In OutputStream, the other two variants will call into write(int b). But to be safe, I will update to add all.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 15
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 05 Feb 2020 02:17:51 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 6:
(9 comments)
Didn't look over the tests yet, but here are a couple high-level suggestions and some nits.
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java@45
PS6, Line 45: BasicSubprocessor
nit: there's not other subprocessor type, so maybe just call this Subprocessor
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java@96
PS6, Line 96: catch (IOException e) {
: LOG.error("Unable to close the underlying output stream", e);
: }
Will this ever catch anything if we're doing System.exit(1) when we hit an exception?
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@44
PS6, Line 44:
: MessageProcessor(long maxMessageBytes) {
: this.maxMessageBytes = maxMessageBytes;
: }
:
: MessageProcessor(long maxMessageBytes,
: BufferedInputStream in) {
: Preconditions.checkNotNull(in);
: this.maxMessageBytes = maxMessageBytes;
: this.in = in;
: }
:
: MessageProcessor(long maxMessageBytes,
: BufferedOutputStream out) {
: Preconditions.checkNotNull(out);
: this.maxMessageBytes = maxMessageBytes;
: this.out = out;
: }
nit: Why do we need so many constructors if we have MessageProcessor(long maxMessageBytes), setInputStream(), and setOutputStream()?
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@52
PS6, Line 52: this.maxMessageBytes = maxMessageBytes;
: this.in = in;
: }
:
: MessageProcessor(long maxMessageBytes,
: BufferedOutputStream out) {
: Preconditions.checkNotNull(out);
: this.maxMessageBytes = maxMessageBytes;
: this.out = out;
: }
:
: MessageProcessor(long maxMessageBytes,
: BufferedInputStream in,
: BufferedOutputStream out) {
: Preconditions.checkNotNull(in);
: Preconditions.checkNotNull(out);
: this.maxMessageBytes = maxMessageBytes;
: this.in = in;
: this.out = out;
nit: these could use setInputStream() and setOutputStream()
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@92
PS6, Line 92: * @throws IllegalStateException if the message is malformed
No longer true?
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@98
PS6, Line 98: interrupted = true;
If we are interrupted, but we try and eventually succeed at putting the message on the queue, will we still break out of the larger loop and interrupt the caller?
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@104
PS6, Line 104: Restore
nit: "Return"? "Propagate"?
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@41
PS6, Line 41: IllegalArgumentException
InvalidProtocolBufferException?
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@69
PS6, Line 69: String confFilePath = cmd.getOptionValue(CONF_LONG_OPTION);
Requiring a configuration file means that Kudu deployments that want to use this will need to create a configuration file and somehow pass that into Kudu. Given that the arguments we're passing here are relatively simple, how do you feel about passing in num writers, queue size, and max bytes as arguments instead?
Eventually I think we'll need config files (e.g. for Ranger configurations), but I don't think the subprocess arguments need to abide the same file-based configuration -- we'd just pull in ranger-site.xml or something.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 6
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 16 Dec 2019 21:30:08 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 15:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@104
PS15, Line 104: if any exceptions occur, no partial
: * message will be written to the underlying output stream
I'm curious how is it possible to guarantee thhis? Does it mean the buffer size is always greater than the buffer in BufferedOutputStream?
I.e., what if the message is 100MB in size, and the size of the buffer in BufferedOutputStream is 10MB, and some IO error happens on writing 10th chunk of the message. Does it mean the first 90MB are not written to the underlying output stream?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 15
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 05 Feb 2020 02:29:09 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#15).
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2971: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate over a stdin/stdout
pipe via protobuf-based protocol. It is useful in cases a Kudu process
(e.g. master) needs to talk to third-party libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal and causes the program to exit, e.g. I/O errors
when reading/writing to the pipe, and parsing malformed protobuf messages.
If encounter InterruptedException during placing/getting messages to/from
the queue, we consider it to be a signal to shutdown the task which
cause the program to exit as well.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoSubprocessMain.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
M java/settings.gradle
16 files changed, 1,410 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/15
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 15
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 5:
(36 comments)
http://gerrit.cloudera.org:8080/#/c/14329/4//COMMIT_MSG
Commit Message:
PS4:
> Could you describe the expectations w.r.t error handling? What kind of "err
Done
http://gerrit.cloudera.org:8080/#/c/14329/4//COMMIT_MSG@22
PS4, Line 22: reading/writing to the pipe
> Seems you also have do update the SubprocessConfiguration
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java@29
PS4, Line 29: @InterfaceAudience.Private
> I like the use of generics here. Could we extend it even further so that we
I don't quite understand what will your proposal look like. Do you mean to have different java programs for each protocol type and in C++ side will run the subprocess based on a configuration? In this case, we need to keep multiple jars?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java@32
PS4, Line 32: private static final String ECHO_MESSAGE = "
> nit: final?
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@35
PS4, Line 35:
:
:
:
:
> nit: maybe note the concurrency expectations of this method, e.g. that we c
Removed BufferedOutputStream in this class.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@47
PS4, Line 47:
> yeah it should be local non-atomic so we retry 3 times for each message ins
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@67
PS4, Line 67:
> we should probably split this method into 3 methods: take from queue, proce
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@69
PS4, Line 69:
> you can use try with resources here. If you do `try (BufferedOutputStream o
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@70
PS4, Line 70:
> this should be `while (!Thread.currentThread().isInterrupted())` instead. T
As mentioned below, we would want to retry with InterruptedException for taking elements from the blocking queue. I don't think `while (!Thread.currentThread().isInterrupted())` is appropriate.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@77
PS4, Line 77:
> We shouldn't catch this here, we don't want to retry anything if the thread
We are taking element from a blocking queue, so we would want to retry when interruption is detected (there is no places in the code to intentionally interrupt the thread), rather than immediately catch and return.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@94
PS4, Line 94:
> I think we shouldn't respond here as we don't have a sequence ID so the ser
Message without sequence ID should be handled in the server(C++ side), the original request will time out as you suggested. I think it is good to at least returned a error message back to the server to indicate something is wrong.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@115
PS4, Line 115:
:
:
:
:
> This never gets reset to false, even if we were eventually table to take fr
Hmm, interrupted is initiated as false and only set to true of an InterruptedException is caught. As mentioned in 'Java Concurrency in Practice' Chapter 7 (p143). It is a good practice to 'Restore the interruption status so that code higher up on the call stack can deal with it'.
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@160
PS1, Line 160:
> Wouldn't it be still better to instantiate a separate MessageProcessor for
Yeah, I agree after thinking a bit more. Updated it.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@39
PS4, Line 39: class MessageProcessor {
> Why does this need to be a class? Couldn't it be a set of utility functions
I decided to wrap inputstream and outputstream as MessageProcessor class member as you suggested previously. So keep it as a class.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@40
PS4, Line 40:
: private long maxMessageBytes;
: private BufferedInputStream in;
: private BufferedOutputStream out;
:
: MessageProcessor(long maxMessageBytes) {
: this.maxMessageBytes = maxMessageBytes;
: }
> Remove this?
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@50
PS4, Line 50: BufferedInputStream
> nit: probably don't need to define a constant for this
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@75
PS4, Line 75:
> or just `return "";`
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@93
PS4, Line 93: // Read four bytes of the message to get the size of the body.
: byte[] sizeBytes = new byte[Integer.BYTES];
: int read = in.read(sizeBytes, 0, Integer.BYTES);
:
> Why have this when writeBinaryMessage already exists?
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@113
PS4, Line 113: * stream. The write is atomic, that is if any exceptions occur, no
: * partial message should be writte
> nit: maybe note that this isn't threadsafe and that we expect there to be a
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@126
PS4, Line 126: // Always do a flush after write to ensure no partial message is written.
: out.flush();
: }
:
: /**
> Can we also include 'read' and 'size' in the error messages to facilitate d
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@148
PS4, Line 148: */
> this should be synchronized. Alternatively (probably a better approach) wou
The write is synchronized in BufferedOutputStream.
I don't see a must have reason to have two blocking queues. I don't see much performance implication with current approach as we are using pipe(which should be fast). But let me know if you think otherwise. Thanks!
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@39
PS4, Line 39:
:
> you could instantiate a bunch of MessageProducers in a thread pool.
Yeah, as what Attila suggested.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@70
PS4, Line 70:
> same as MessageConsumer
Ack
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@71
PS4, Line 71:
> same as MessageConsumer
Ack
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@88
PS4, Line 88:
> same as MessageConsumer, we don't have a sequence ID so we should just log
Ack
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@98
PS4, Line 98:
> Why aren't we responding with an error here?
I added a warning as I don't see it is necessary an error (the message can be parsed). But let me know if you think otherwise.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@100
PS4, Line 100:
> same as MessageConsumer
Ack
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@106
PS4, Line 106:
> we shouldn't retry after an interruption, as in this case the expected beha
Ack
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@37
PS4, Line 37: SubprocessResponseP
> SubprocessResponsePB
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@64
PS4, Line 64: abstract Class<Request> getRequestClazz();
> Will this always be:
I don't think you can create an instance of a generic type, so 'return Request.class;' is not possible.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@95
PS4, Line 95: }
> I think this should be long just in case. In the C++ side we use uint but a
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@63
PS4, Line 63: ExecutorService producerService = Executors.newSingleThreadExecutor();
> maybe we can use try with resources here and then join the threads at the e
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@80
PS4, Line 80:
> there should be a separate consumer instance for each thread.
I don't see why it is necessary to have separate consumer instance for each thread?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@88
PS4, Line 88: consumerFuture.exceptionally(errorHandler);
> don't think these are needed, shutdown only means the executor service won'
I agree this is not needed, and as in each CompletableFuture has error handling for any exceptions. I don't think shutdownNow() is needed as well. Let me if you think I miss anything.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java@42
PS4, Line 42: @RunWith(Parameterized.class)
: public class TestMessageProcessor e
> There's only one serialization mode now.
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java@78
PS4, Line 78:
> this should be in its own class, but as Andrew said, we probably don't need
Done
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 5
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 09 Dec 2019 17:40:39 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] process communicates via protobuf-based protocol
......................................................................
Patch Set 1:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG@14
PS1, Line 14: reader(producer)
Could we just call them reader and writer then? Producer and consumer is a little unintuitive IMO because when I'm writing, I'm producing, and when I'm reading, I'm consuming.
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG@15
PS1, Line 15: (with optional JSON encoding)
Can you elaborate on why this is useful? The main use case I know of is for structured, schema-ed messages, which makes protobuf a natural fit. Why bother with all the extra code for JSON?
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java@17
PS1, Line 17: // Licensed to the Apache Software Foundation (ASF) under one
: // or more contributor license agreements. See the NOTICE file
: // distributed with this work for additional information
: // regarding copyright ownership. The ASF licenses this file
: // to you under the Apache License, Version 2.0 (the
: // "License"); you may not use this file except in compliance
: // with the License. You may obtain a copy of the License at
: //
: // http://www.apache.org/licenses/LICENSE-2.0
: //
: // Unless required by applicable law or agreed to in writing,
: // software distributed under the License is distributed on an
: // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
: // KIND, either express or implied. See the License for the
: // specific language governing permissions and limitations
: // under the License.
Delete
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@42
PS1, Line 42: static int retries
Sorry if this is a newbie java question, but doesn't this mean that this is shared among all instantiations of MessageConsumer?
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@160
PS1, Line 160: BufferedInputStream in
Why bother passing these around? Why not just make them members of MessageProcessor?
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@95
PS1, Line 95: retries
Should this be local to the MessageProducer rather than static?
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@31
PS1, Line 31: Responses a protobuf request message
nit: "Responds to a protobuf request message."
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@38
PS1, Line 38: public class SubprocessCommandLineParser {
Maybe a newbie java question, but why go through the trouble of building this out as a tool with CLI arguments and a configuration file, versus running a jar and using system properties?
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProducerConsumer.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProducerConsumer.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProducerConsumer.java@131
PS1, Line 131: MessageConsumer consumer = new MessageConsumer(blockingQueue, messageProcessor, out);
: for (int i = 0; i < CONSUMER_TASK_NUM; i++) {
: CompletableFuture consumerFuture =
: CompletableFuture.runAsync(consumer, consumerService);
: consumerFuture.exceptionally(errorHandler);
: }
It'd be nice if we had a test that stressed this with many input messages and a bit of slowness in the processing to see what happens if the consumers try to write in parallel.
http://gerrit.cloudera.org:8080/#/c/14329/1/src/kudu/subprocess/subprocess.proto
File src/kudu/subprocess/subprocess.proto:
http://gerrit.cloudera.org:8080/#/c/14329/1/src/kudu/subprocess/subprocess.proto@24
PS1, Line 24: message EchoRequestPB {
: required string data = 1;
: }
: message EchoResponsePB {
: required string data = 1;
: }
Are java generics not expressive enough to allow us to just use EchoRequestPB/EchoResponsePB directly (if we put the AppStatusPB error into these messages), rather than wrapping with a SubprocessRequestPB?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 1
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 01 Oct 2019 06:45:38 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14329/8/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/8/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@79
PS8, Line 79:
> The problem is in the test the input stream is short, the reader task can g
have you tried that? I think EOF has to be specifically sent to get an EOFException. If you don't close the stream or send EOF it will just return the bytes read and unblock. So you should be able to interrupt the thread, then write something to the stream and it would guarantee that it is interrupted when blockingQueue.put() is reached.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 10
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 16 Jan 2020 23:24:49 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#4).
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2791: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate with other process
via protobuf-based protocol over standard input/output. It is useful
in cases a Kudu process (e.g. master) needs to talk to third-party
libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface to specify how to handle the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
M java/gradle/dependencies.gradle
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLine.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProducerConsumer.java
M java/settings.gradle
17 files changed, 1,385 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/4
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 4
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#19).
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2971: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate over a stdin/stdout
pipe via protobuf-based protocol. It is useful in cases a Kudu process
(e.g. master) needs to talk to third-party libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal and causes the program to exit, e.g. I/O errors
when reading/writing to the pipe, and parsing malformed protobuf messages.
If encounter InterruptedException during placing/getting messages to/from
the queue, we consider it to be a signal to shutdown the task which
cause the program to exit as well.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess-echo/build.gradle
A java/kudu-subprocess-echo/src/main/java/org/apache/kudu/subprocess/echo/EchoProtocolHandler.java
A java/kudu-subprocess-echo/src/main/java/org/apache/kudu/subprocess/echo/EchoSubprocessMain.java
A java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
M java/settings.gradle
17 files changed, 1,414 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/19
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 19
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 7:
(29 comments)
http://gerrit.cloudera.org:8080/#/c/14329/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/7//COMMIT_MSG@9
PS7, Line 9: other
Should clarify here that because communication takes place over a stdin/stdout pipe, "other" really means whichever process spawns the Java process.
http://gerrit.cloudera.org:8080/#/c/14329/7//COMMIT_MSG@18
PS7, Line 18: 2) multiple writer threads that continuously retrieve the messages
: from the queue, process them and write the responses to the standard
: output.
Curious why you didn't go with writing the messages to another blocking queue, and using a separate thread to then poll that queue and write its messages to stdout. The symmetry of such an approach with #1 could mean more reusable code.
Something to consider: if stdout fills up (because the spawning process isn't consuming it fast enough), anyone who tries to write to it will block. With the current design, that means blocking the writing threads and request processing. With the change I'm proposing, only the single "responding" thread blocks and the other threads are free to continue processing requests and posting the responses to the second blocking queue. That is, instead of blocking all request processing, we'll just add more and more items to the queue.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@72
PS7, Line 72: if (in.available() <= 0) {
: return "";
: }
Why do we do this? Don't we want to block waiting for additional input if none exists? Looks like MessageReader will spin in a tight loop in this case anyway, which doesn't seem right.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@83
PS7, Line 83: "message size (" + size + ") exceeds maximum message size (" +
: maxMessageBytes + ")"
Nit: reformat using String.format
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@90
PS7, Line 90: "unable to receive message body");
Maybe include how much was actually read?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@91
PS7, Line 91: return new String(dataBytes, StandardCharsets.UTF_8);
Why are we converting this into a UTF-8 String? It's a serialized PB message, right? Shouldn't it be a byte array?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@122
PS7, Line 122: String
Why is this a String? Shouldn't it be a byte array?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@133
PS7, Line 133: ByteBuffer buffer = ByteBuffer.wrap(data)
: .order(ByteOrder.BIG_ENDIAN);
: return buffer.getInt();
Nit: could combine:
return ByteBuffer.wrap(...).order(...).getInt();
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@145
PS7, Line 145: ByteBuffer buffer = ByteBuffer.allocate(Integer.BYTES)
: .order(ByteOrder.BIG_ENDIAN)
: .putInt(value);
: return buffer.array();
Nit: could combine:
return ByteBuffer.allocate(...).order(...).putInt(...).array();
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@44
PS7, Line 44: private static int RETRY_NUM = 3;
Should be final too.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@60
PS7, Line 60: If IllegalStateException occur,
: // which indicates the message is malformed, respond with an error message
: // and continue to the next element on the queue.
How is this safe? AFAICT, in most cases that readBytes() throws IllegalStateException, there's a good chance that it didn't fully process a message from the pipe, which would interfere with any future calls to readBytes. Nor could it process it: the two ends of the pipe aren't speaking the same language.
I guess what I'm saying is this seems like it should be a fatal error.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@84
PS7, Line 84: // InterruptedException, record the interruption status and retry.
Why do we retry if we've been interrupted? Isn't interruption a sign that the subprocess should exit?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@98
PS7, Line 98: interrupted = true;
IIUC, interruption means we retry the put() but then exit the main loop. Why do we bother with the put at all in this case?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@60
PS7, Line 60: public void run() {
Same questions about interruption as for MessageReader.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@130
PS7, Line 130: * message write is atomic, retries on IOException until reaching the limits.
I don't understand why we retry here at all, or what limits we're talking about. If stdout is full, attempts to write to it will block. And that's fine, isn't it?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@64
PS7, Line 64: getRequestClazz
Nit: the 'clazz' variant is only necessary for variable names, where 'class' is a reserved word. You can use 'class' inside a function name though; getRequestClass is perfectly legal.
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@69
PS6, Line 69: String confFilePath = cmd.getOptionValue(CONF_LONG_OPTION);
> I'm imagining how this would work end-to-end and using config files, I thin
I agree with Andrew that the configuration for the _subprocess itself_ ought to be expressed as arguments rather than a file. Here's another argument: you already wrote some code to parse command line args just to extract the config file arg; why not reuse that code to parse additional args and drop the config file loading stuff?
BTW, I expect that subprocess (and Ranger and Atlas and ...) configuration will be fully managed by the Kudu master. If not, we can't rev "both sides" and we suddenly have to worry about backwards compatibility across this interface.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@60
PS7, Line 60: getMaxWriterThread
Nit: this (and associated fields) should be plural i.e. maxWriterThreads.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Subprocessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Subprocessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Subprocessor.java@69
PS7, Line 69: true
Nit: add /* fair= */ here.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Subprocessor.java@79
PS7, Line 79: try (BufferedInputStream in = new BufferedInputStream(System.in);
Do we need to need to join on all of these futures? Via something like CompletableFuture.allOf()?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/resources/log4j2.properties
File java/kudu-subprocess/src/main/resources/log4j2.properties:
PS7:
Every other log4j2.properties file also has this section:
logger.kudu.name = org.apache.kudu
logger.kudu.level = debug
What does it do? Should we add it here for consistency?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@33
PS7, Line 33: public class MessageTest {
Could you rename this to make it more clear that it's a test fixture? Maybe "BaseMessageTest"?
Also, could you add some docs?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@38
PS7, Line 38: * Construct a SubprocessRequestPB message based on the given request
Nit: for new files, try to use indicative tense ("Constructs a ...") vs. an imperative tense ("Construct a ..."). For existing files it's best to just match the existing style.
There's more on this in the Google Style Guide (https://google.github.io/styleguide/cppguide.html#Function_Comments).
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@45
PS7, Line 45: get
Nit: maybe 'create' or 'build' would be more precise?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@46
PS7, Line 46: String payload, String messageType) {
I don't understand how this method is going to be extensible for other message types. It's very likely that only the echo message type will have a single "payload" argument; other message types will no doubt be richer and accept more arguments.
If extensibility isn't a goal, perhaps rename it to createEchoSubprocessRequest, drop the message type arg, and special-purpose it as a test-only method for echo messages?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@68
PS7, Line 68: getSerializedMessage
Nit: maybe just serializeMessage?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@78
PS7, Line 78: * Read the given message in byte array from a output stream and returns
Sort of expected this to be more symmetric with getSerializedMessage. Maybe I'll understand why they're not if I keep reading...
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@86
PS7, Line 86: <T extends Message> T getDeserializedMessage(byte[] bytes,
Nit: and deserializeMessage?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java:
PS7:
New tests should include the RetryRule so that they properly retry on failure (if flaky) and report their test results.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 7
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 18 Dec 2019 19:32:08 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14329/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/7//COMMIT_MSG@7
PS7, Line 7: KUDU-2791
KUDU-2971
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 7
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 09 Jan 2020 14:08:58 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 5:
(4 comments)
> Patch Set 5:
>
> (8 comments)
Sorry, missed your replies. Went back and expanded on them.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@70
PS4, Line 70:
> As mentioned below, we would want to retry with InterruptedException for ta
hm... there's no way to gracefully shut down in this case though. If we don't want to interrupt the thread at all, why do we need the whole interrupt logic at all?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@148
PS4, Line 148: */
> The write is synchronized in BufferedOutputStream.
even if BufferedOutputStream synchronizes, write and flush, I think it should still be in the same synchronization block. Also, if we ever replace BufferedOutputStream with another OutputStream it would be an issue and even if the methods are synchronized on BufferedOutputStream, it doesn't say anything about thread safety in its contract so theoretically there could be a JVM implementation where this is not the case.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@80
PS4, Line 80:
> I don't see why it is necessary to have separate consumer instance for each
yeah I think you're right and it's not needed.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@88
PS4, Line 88: consumerFuture.exceptionally(errorHandler);
> I agree this is not needed, and as in each CompletableFuture has error hand
yeah we can skip shutdowns here, but maybe add them in a shutdown hook?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 5
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 14 Dec 2019 02:25:05 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 7:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java@96
PS6, Line 96:
:
:
> Are you asking if close() will be called with System.exit(1)? The oracle Ja
Ah, I was asking how this interacts, if at all, with ERROR_HANDLER, but based on your explanation, it seems this is mostly for catching errors with 'in' and 'out'.
Also, are we guaranteed that the error came from the output stream? Could it have come from the input stream?
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@92
PS6, Line 92: }
> This is refer to L106 and L113
Ah, didn't realize Preconditions meant we didn't have to annotate with 'throws IllegalStateException'. Thanks for explaining.
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@98
PS6, Line 98: interrupted = true;
> Are you asking should we do that? I don't quite understand why we still wan
I'm asking whether that's what we're doing. 'finally' blocks are run unconditionally, so if we _were_ interrupted, but we retried and eventually succeeded, won't this still interrupt the caller? Is that desired?
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@69
PS6, Line 69: String confFilePath = cmd.getOptionValue(CONF_LONG_OPTION);
> I feel configuration file is more flexible if we have more (or less) config
I'm imagining how this would work end-to-end and using config files, I think, means either:
- That the caller of this tool (ie. the Kudu master) needs to generate a config file for each subprocess, based on gflags that are passed to the master. Or,
- Cloudera Manager (or equivalent) would need to generate a config file based on some configuration through CM, for each subprocess service.
Yes this is _similar_ to what we do for Sentry and the HMS, but it's different in that this would have to be done for every subprocess, meaning we would have a config file for Ranger, a config file for Atlas, a config file for the Kudu-Ranger subprocess, _and_ a config file for the Kudu-Atlas subprocess -- it just seems like a lot.
Contrast that to using command-line arguments for the binary -- the caller (ie the Kudu master) would run the tool, passing arguments directly to the binary, and we'd only need the Ranger and Atlas config files.
I think these are all equally flexible, but having command-line arguments seems like the path of least resistance here, and the least unwieldy IMO.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 7
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 17 Dec 2019 04:10:16 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 12:
(13 comments)
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@39
PS10, Line 39:
> nit: there isn't a whole lot of processing being done in this class. It see
Done
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@130
PS10, Line 130:
> Can this be static? Why is this in this class?
As this is one line code method, I removed it as it doesn't give much value have it.
http://gerrit.cloudera.org:8080/#/c/14329/8/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/8/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@79
PS8, Line 79: break;
> have you tried that? I think EOF has to be specifically sent to get an EOFE
Yeah, I have tried that, with ByteArrayInputStream whenever finish reading the input byte array, it considers the end of stream reached and returns '-1'.
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@50
PS10, Line 50:
> nit: since we're expecting there to eventually be an outbound queue, maybe
Done
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@46
PS10, Line 46:
> nit: same here, maybe rename this to inboundQueue? Especially if the Messag
Done
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@48
PS10, Line 48: static final String WRITE_MSG_ERR = "Unable
> nit: not really sure what protocol is referring to here. Maybe call this Re
Makes sense, updated it to ProtocolHandler.
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@107
PS10, Line 107: sponse(byte[] data) {
: SubprocessResponsePB response;
> This doesn't use any of the MessageProcessor's internal state, so maybe jus
Done
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@44
PS10, Line 44:
> nit: maybe call this handleSubprocessRequest() or something?
Done
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@58
PS10, Line 58:
> nit: this doesn't actually send a response; maybe rename it and update this
Done
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@63
PS10, Line 63:
> nit: maybe call this handleRequest() or something?
I updated it to 'createResponse'.
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@40
PS10, Line 40: THREAD
> nit: this should be plural
Done
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java@60
PS10, Line 60:
> This doesn't look like it's creating a message that is longer than 1024*102
Done
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java@142
PS10, Line 142:
:
:
:
:
:
> I might be missing something here. Mind adding a comment what we're testing
Done
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 12
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 27 Jan 2020 06:05:54 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#6).
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2791: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate with other process
via protobuf-based protocol over standard input/output. It is useful
in cases a Kudu process (e.g. master) needs to talk to third-party
libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal if encountered when reading/writing to the pipe
and can cause the tool to exit, while an error message is responded
for malformed protobuf message.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProcessorMain.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLine.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java
M java/settings.gradle
17 files changed, 1,386 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/6
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 6
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 2: Verified+1
Unrelated flaky test.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 14 Oct 2019 18:36:18 +0000
Gerrit-HasComments: No
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 2:
(3 comments)
Haven't reviewed yet, just responding to comments.
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG@15
PS1, Line 15: optional JSON encoding) from
> I think this is mainly a match up with the C++ side protobuf format support
Why would we want to send JSON over the channel though?
If it's for better debugging, why not implement pretty printers on the Java side (e.g. https://github.com/apache/kudu/blob/0f2946b74ea138b29b00f68047c50f44674ddc5a/java/kudu-client/src/main/java/org/apache/kudu/client/ProtobufHelper.java)? And C++ has pb_util that support printing JSON already.
I really don't see a really compelling reason keep JSON as a serialization format, and it's a large part of this patch.
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@38
PS1, Line 38: public class SubprocessCommandLineParser {
> I think it is more specific in this way, as there are some configurations t
But can't you pass in arguments to a Java program as system properties? E.g.
https://stackoverflow.com/questions/5045608/proper-usage-of-java-d-command-line-parameters
Don't those properties only get used by that single Java process?
http://gerrit.cloudera.org:8080/#/c/14329/1/src/kudu/subprocess/subprocess.proto
File src/kudu/subprocess/subprocess.proto:
http://gerrit.cloudera.org:8080/#/c/14329/1/src/kudu/subprocess/subprocess.proto@24
PS1, Line 24: message EchoRequestPB {
: required string data = 1;
: }
: message EchoResponsePB {
: required string data = 1;
: }
> I guess so, but what is the intention behind? Moreover, with the current ap
message EchoRequestPB {
optional int64 id = 1;
optional string data = 2;
}
message EchoResponsePB {
optional int64 id = 1;
optional AppStatusPB error = 2;
optional string data = 3;
}
This seems cleaner than having to add a 1) new message and updating 2) SubprocessRequestPB and 3) SubprocessResponsePB. It seems more aligned with the existing code style too, where the underlying protobuf message is the first-class citizen, not the communication channel. For instance, if you look at rpc/rpc_header.proto, we do basically what I'm suggesting with the definition of RequestHeader and ResponseHeader.
Also I envision this feature being the addition of binaries along the lines of:
java -Dfoo="bar" kuduRangerServer.jar
where the Message* classes here are templatized to process specific message types, and the kuduRangerServer has a MessageServer<RangerRequestPB, RangerResponsePB> that encapsulates all the Message* work and doesn't have any switch statements on the message type (because we already know we're handling Ranger requests). I'm not sure what the benefit is of allowing the Message* classes to parse multiple kinds of messages with this oneof switching.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 15 Oct 2019 01:38:42 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 20: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 20
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 07 Feb 2020 20:26:28 +0000
Gerrit-HasComments: No
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 12:
(20 comments)
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java@48
PS12, Line 48: Executor
nit: FWIW I think Subprocess for this was fine, considering that's what we call it elsewhere (KuduSubprocessException, SubprocessConfiguration, etc.). It's a bit more confusing now because java.util.concurrent.Executors exists here too.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java@50
PS12, Line 50: private Function<Throwable, Object> errorHandler;
Can this be static final too?
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java@88
PS12, Line 88: // need to 1. ensure no partial message write due to auto flush
: // 2. PrintStream is sawlloing IOexception
nit: so it's clear you're referring to the SubprocessOutputStream, maybe instead say:
"Wrap the system output in a SubprocessOutputStream so IOExceptions from system output are propagated up instead of being silently swallowed."
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java@112
PS12, Line 112: else {
: readerFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
: CompletableFuture.allOf(writerFutures)
: .get(timeoutMs, TimeUnit.MILLISECONDS);
: }
I'll probably get to this later, but why would we impose a time limit on the entire runtime of the subprocess? Aren't they meant to be long-running?
Yeah, seems like this is only used in tests to ensure our tests don't run forever. Maybe comment with something to the effect of:
"In cases where we don't want our executor to run forever, e.g. in tests, wait for the specified timeout."
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java@29
PS12, Line 29: subprocess
Remove?
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@53
PS12, Line 53: The read is not atomic (partial read can happen if any
: * exceptions occur) and blocking (waits until the input is available).
nit: maybe be more prescriptive here? E.g.
"The read is blocking and not atomic (partial reads can occur if exceptions occur). As such, users should ensure this is not called from multiple threads concurrently."
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@103
PS12, Line 103: The write is atomic, that is if any exceptions occur, no partial
: * message should be written to the underlying output stream.
Is it possible that we'll raise an exception in flush(), and then we're left with some bytes written but not flushed? I guess we wouldn't ever be able to flush since subsequent calls to writeMessage() will raise exceptions after more calls to write().
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@113
PS12, Line 113: this
nit: at first I thought we might want to synchronize on 'out' instead of 'this'. But according to https://www.geeksforgeeks.org/synchronized-in-java/ the effect on concurrency would be the same since these are the only synchronized blocks in this class. That said, maybe we ought to synchronize on 'out' so it's more obvious we're only worried about blocking concurrent writes?
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@91
PS12, Line 91: respondWithError
nit: this doesn't actually send the response so it's not "responding" per se. Maybe "responseWithError" since this just creates the response?
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java@25
PS12, Line 25: * The {@link SubprocessOutputStream} class is a wrapper around {@link PrintStream}
: * for explicitly re-throw <code>IOException</code> if any encountered in a
: * <code>PrintStream</code>. Because unlike other output streams, a <code>PrintStream</code>
: * never throws an <code>IOException</code>; instead, exceptional situations
: * merely set an internal flag that can be tested via the <code>checkError</code>
: * method.
nit: looking around for other wrappers, maybe reword a bit?
Wrapper around {@link java.io.PrintStream} that throws an <code>IOException</code> instead of relying on explicit <code>checkError</code> calls when writing or flushing to the stream. This makes its error-throwing behavior more similar to most {@link java.io.OutputStream}s.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java@47
PS12, Line 47: @Override
nit: spacing
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java@49
PS12, Line 49: out.flush();
Seem like we might not need this?
"Flushes the stream and checks its error state."
from https://docs.oracle.com/javase/7/docs/api/java/io/PrintStream.html#checkError()
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@90
PS12, Line 90: Assert.assertTrue(false);
Could we also display the error here?
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@156
PS12, Line 156: String[] args = {"-w", "1"};
Is this important for this test?
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@194
PS12, Line 194: if needed.
remove this
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@195
PS12, Line 195: reader.interrupt();
Rather than copying the guts of Executor in this test, could we instead plumb some interrupt() into the Executor class itself? Then this test would look a bit more like the other tests, and we wouldn't need to reimplement this test if the guts of Executor ever change.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@214
PS12, Line 214: CompletableFuture.anyOf(writerFutures)
: .get(1000, TimeUnit.MILLISECONDS);
Can't we wait for all of them to finish? If there's nothing in the pipe, won't these just time out?
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@39
PS12, Line 39: public class TestMessageIO {
How difficult would it be to plumb a PrintStream down and then call setError() so we can test the exception handling on the write side?
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@58
PS12, Line 58: // Construct a message that it is not properly serialized with the message size
: // that causes maximum message size exceed exception.
: byte[] malformedMessage = "malformed message".getBytes(StandardCharsets.UTF_8);
Can you be more explicit and explain the UTF-8 will expand this message to be more bytes than is expected? Or how about explicitly using a string that is larger than the buffer size? It can be a bit surprising to see "malformed message" be considered an exceedingly large message.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@75
PS12, Line 75: byte[] body = "malformed message".getBytes(StandardCharsets.UTF_8);
Does it make sense to add a test for what happens if there isn't enough data in the pipe?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 12
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 27 Jan 2020 23:11:49 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@72
PS7, Line 72: String.format("message size (%d) exceeds maximum message size (%d)",
: size, maxMessageBytes));
: }
> As InputStream doesn't throw InterruptedException when it's interrupted (it
I am not sure in which scenarios you would want to interrupt/cancel the blocking if not encounter I/O error or end of stream?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 8
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 14 Jan 2020 19:15:52 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 9:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@84
PS7, Line 84: // InterruptedException during the put, record the interruption
> If the thread is interrupted, we should terminate. If the blocking queue is
We discussed this offline, and decide to exit if encounter InterruptedException (consider it is a signal to shutdown the task, since we do not have intentionally interrupt call in the code). We also decided to remove the shutdown hook in Subprocessor as gracefully shutdown is not essential as we don't maintain states in the Subprocess.
http://gerrit.cloudera.org:8080/#/c/14329/8/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/8/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@79
PS8, Line 79: if (injectInterrupt) {
> I'm not sure I understand, the thread would be blocking on either messagePr
The problem is in the test the input stream is short, the reader task can get EOF before being interrupted by the test thread.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/resources/log4j2.properties
File java/kudu-subprocess/src/main/resources/log4j2.properties:
PS7:
> Answering my own question: unlike our other modules (which basically provid
Ack
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 9
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 15 Jan 2020 23:35:40 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 16:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java:
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@51
PS15, Line 51: private static Function<Throwable, Object> ERROR_HANDLER = (t) -> {
> Oh, I missed that usage. In that case, please move this initialization back
Done
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@59
PS15, Line 59: SubprocessExecutor() {
: }
> Right, I missed that there's another constructor. My bad.
Ack
http://gerrit.cloudera.org:8080/#/c/14329/16/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java:
http://gerrit.cloudera.org:8080/#/c/14329/16/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@47
PS16, Line 47: public RetryRule retryRule = new RetryRule();
> Does this still need a @Rule annotation? Or not when paired with a RuleChai
Yeah, I think it is not needed here when paired with RuleChain.
http://gerrit.cloudera.org:8080/#/c/14329/16/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/16/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@45
PS16, Line 45: public RetryRule retryRule = new RetryRule();
> Same question.
Same here.
http://gerrit.cloudera.org:8080/#/c/14329/16/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@116
PS16, Line 116: thrown.expect(IOException.class);
: thrown.expectMessage("exceeds maximum message size");
: messageIO.readBytes();
> This works mid-test? I thought the call that throws an exception would have
Ah, sorry that actually it is not working... the test exits in the middle as what you understand. Updated it be another test.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 16
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 06 Feb 2020 05:43:04 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#12).
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2971: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate over a stdin/stdout
pipe via protobuf-based protocol. It is useful in cases a Kudu process
(e.g. master) needs to talk to third-party libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal and causes the program to exit, e.g. I/O errors
when reading/writing to the pipe, and parsing malformed protobuf messages.
If encounter InterruptedException during placing/getting messages to/from
the queue, we consider it to be a signal to shutdown the task which
cause the program to exit as well.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoSubprocessMain.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
M java/settings.gradle
17 files changed, 1,397 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/12
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 12
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 10:
(13 comments)
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@39
PS10, Line 39: MessageProcessor
nit: there isn't a whole lot of processing being done in this class. It seems squarely focused on reading from the pipe and writing to the pipe. Maybe rename this MessageIO or something?
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@130
PS10, Line 130: <T extends Message> T bytesToMessage(byte[] data, Parser<T> parser)
Can this be static? Why is this in this class?
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@50
PS10, Line 50: blockingQueue
nit: since we're expecting there to eventually be an outbound queue, maybe call this inboundQueue or something?
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@46
PS10, Line 46: blockingQueue
nit: same here, maybe rename this to inboundQueue? Especially if the MessageWriter will interact with the outboundQueue.
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@48
PS10, Line 48: private ProtocolProcessor protocolProcessor;
nit: not really sure what protocol is referring to here. Maybe call this RequestHandler requestHandler?
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@107
PS10, Line 107: messageProcessor.bytesToMessage(
: data, SubprocessRequestPB.parser());
This doesn't use any of the MessageProcessor's internal state, so maybe just inline the call here?
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@44
PS10, Line 44: Subprocess.SubprocessResponsePB processRequest(Subprocess.SubprocessRequestPB request)
nit: maybe call this handleSubprocessRequest() or something?
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@58
PS10, Line 58: Responds
nit: this doesn't actually send a response; maybe rename it and update this to focus on the fact that it's doing stuff with the request?
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@63
PS10, Line 63: responses
nit: maybe call this handleRequest() or something?
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@40
PS10, Line 40: THREAD
nit: this should be plural
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java@60
PS10, Line 60: byte[] malformedMessage = "malformed message".getBytes(StandardCharsets.UTF_8);
This doesn't look like it's creating a message that is longer than 1024*1024 bytes. Mind commenting why this is malformed?
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java@119
PS10, Line 119: // parsing non-malformed message exits normally without any exceptions.
: {"test", false, false, false, false, false, noErrors},
: // parsing empty input message exits normally without any exceptions.
: {"", false, true, false, false, false, noErrors},
: // parsing message with empty payload exits normally without any exceptions.
: {emptyPayload, true, true, false, false, false, noErrors},
: // parsing malformed message causes IOException.
: {"malformed", false, true, true, false, false, hasError},
: // parsing message with IOException injected exits with exception.
: {"test", false, false, false, true, false, hasError},
: // parsing message with InterruptedException injected exits with exception.
: {"test", false, false, false, false, true, hasError},
nit: IMO this makes this test significantly more difficult to read.
Maybe consider instead having a test method that takes as a MessageProcessor, MessageReader, MessageWriter, and a message, and returns the output. And then define multiple test cases that each use this method but set up the inputs with different failure modes, and then inspect the output/exceptions/etc in each test case. I think that'd make it much easier to correspond bad behavior with the expected error.
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java@142
PS10, Line 142: if (!isSerialized) {
: request = MessageTestUtil.createEchoSubprocessRequest(message);
: messageBytes = MessageTestUtil.serializeMessage(request);
: } else {
: messageBytes = message.getBytes(StandardCharsets.UTF_8);
: }
I might be missing something here. Mind adding a comment what we're testing here and why? Like why are we serializing the message isSerialized is false?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 10
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 16 Jan 2020 23:41:30 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 15:
(9 comments)
OK I reviewed the tests too, though with an eye towards reducing boilerplate rather than code coverage, which I expect others have already done.
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@91
PS14, Line 91:
:
:
:
> If I understand it correctly, you are suggesting to combine CommandLine wit
Oh, I just meant that:
CommandLine cmd = parser.parse(options, args);
String queueSize = cmd.getOptionValue(QUEUE_SIZE_LONG_OPTION);
String maxWriterThreads = cmd.getOptionValue(MAX_WRITER_THREADS_LONG_OPTION);
String maxMsgBytes = cmd.getOptionValue(MAX_MESSAGE_BYTES_LONG_OPTION);
conf = new SubprocessConfiguration(queueSize, maxWriterThreads, maxMsgBytes);
Could become:
CommandLine cmd = parser.parse(options, args);
conf = new SubprocessConfiguration(cmd);
And then the SubprocessConfiguration constructor would access 'cmd' to retrieve whatever configuration values it needed.
But I don't have a strong opinion on this; combining SubprocessConfiguration and SubprocessCommandLineParser is fine with me.
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java:
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@51
PS15, Line 51: private static Function<Throwable, Object> errorHandler = (t) -> {
Could be final?
Should be named ERROR_HANDLER.
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@59
PS15, Line 59: SubprocessExecutor() {
: }
Can be removed?
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java@44
PS14, Line 44: @Override
> In OutputStream, the other two variants will call into write(int b). But to
What about close()? Do we need to provide an override for it?
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java:
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@81
PS15, Line 81: @Rule
: public RetryRule retryRule = new RetryRule();
Nit: move this to just after the declaration of LOG so that it's easier to find.
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@104
PS15, Line 104: } catch (TimeoutException e) {
I think you can simplify this code by adding (expected = TimeoutException.class) to the @Test annotation. Then you don't need the try/catch here at all, or the fail(). Same for testMsgWithEmptyPayload.
For the other tests, you'll need to use an ExpectedException rule (https://github.com/junit-team/junit4/wiki/Exception-testing#expectedexception-rule). But in order to avoid unexpected interactions with the RetryRule you'll also need to use a RuleChain. See commit ef900efdc for details.
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@41
PS15, Line 41: read and write
reading and writing
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@43
PS15, Line 43: public class TestMessageIO {
Same comments about using an ExpectedException Rule and a RuleChain.
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@53
PS15, Line 53: @Override
: public void setError() {
: super.setError();
: }
Should add a comment explaining what's happening here: you're expanding the visibility of setError() so that you can call it from tests.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 15
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 05 Feb 2020 18:45:13 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 3:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG@15
PS1, Line 15: optional JSON encoding) from
> Agreed with Andrew and Grant. The "subprocess protocol" module must support
Alright, removed the JSON support based on all of your suggestion.
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/build.gradle
File java/kudu-subprocess/build.gradle:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/build.gradle@26
PS2, Line 26: compile libs.slf4jsimple
> What is slf4jsimple required for?
Done
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java@27
PS2, Line 27: public class EchoProtocolProcessor implements ProtocolProcessor<EchoRequestPB, EchoResponsePB> {
> Should we added Private annotations to all of these classes to make sure no
Done
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@95
PS2, Line 95: for(; retries.intValue() > 0; retries.getAndDecrement()) {
> Maybe use a while loop? A for loop without an index initialization is fair
Done
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@96
PS2, Line 96: continue;
> Is this an exceptional occurrence? When would the message be empty?
Yeah, this is an abnormal case and not expected to happen. Adding the handling in case a correct message is received after a corrupted one.
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@38
PS1, Line 38: public class SubprocessCommandLineParser {
> But can't you pass in arguments to a Java program as system properties? E.g
There could be a bunch of properties (that can change), like the size of the blocking queue, which would be easier to manage with config file.
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@55
PS1, Line 55: static void run(SubprocessConfiguration conf) {
> Could be private? Or if this visibility is only for testing, annotate with
Done
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@41
PS2, Line 41: private static final int QUEUE_SIZE = 100;
> How was this size chosen?
This is a randomly chosen number, but I make it configurable to be able customize it.
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessRuntimeException.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessRuntimeException.java:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessRuntimeException.java@23
PS2, Line 23: final class SubprocessRuntimeException extends RuntimeException {
> Maybe make this clearly Kudu related. `KuduSubprocessException`.
Done
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/resources/log4j.properties
File java/kudu-subprocess/src/main/resources/log4j.properties:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/resources/log4j.properties@1
PS2, Line 1: log4j.debug=true
> Match the log4j2.properties files like in all other modules.
Done
http://gerrit.cloudera.org:8080/#/c/14329/1/src/kudu/subprocess/subprocess.proto
File src/kudu/subprocess/subprocess.proto:
http://gerrit.cloudera.org:8080/#/c/14329/1/src/kudu/subprocess/subprocess.proto@24
PS1, Line 24: message EchoRequestPB {
: required string data = 1;
: }
: message EchoResponsePB {
: required string data = 1;
: }
> Ah I see this is based off of the work in tools/tool.proto.
I moved to the protobuf definition to https://gerrit.cloudera.org/c/14426/ as it is required for both Java and C++ patches. I adopted Adar's suggestion to use Protobuf Any support and your suggestion to templatize the message type.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 3
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 18 Nov 2019 18:10:24 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 14: Verified+1
Unrelated flaky test.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 14
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 04 Feb 2020 17:07:30 +0000
Gerrit-HasComments: No
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#7).
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2791: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate with other process
via protobuf-based protocol over standard input/output. It is useful
in cases a Kudu process (e.g. master) needs to talk to third-party
libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal if encountered when reading/writing to the pipe
and can cause the tool to exit, while an error message is responded
for malformed protobuf message.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProcessorMain.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLine.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Subprocessor.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java
M java/settings.gradle
17 files changed, 1,360 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/7
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 7
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 19:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@33
PS17, Line 33: public class SubprocessConfiguration {
> I want to avoid adding inline comment /*blah=*/, also some of the variables
But with the exception of the boolean, these are all pretty self explanatory. At least, looking at how we define such options and descriptions in the C++ tooling, defining constants for these seems like overkill.
And even if we did want to check the strings match in tests, we usually favor matching on the string literal rather than a constant (see many of our C++ tooling tests for examples) -- we're testing that we get a message that makes sense for the test, not that we're using a specific string variable.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 19
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 07 Feb 2020 07:49:53 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 18:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/build.gradle
File java/kudu-subprocess/build.gradle:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/build.gradle@46
PS17, Line 46: jar {
: manifest {
: attributes(
: 'Main-Class': 'org.apache.kudu.subprocess.EchoSubprocessMain'
: )
: }
: }
> Hrm, should we ship all the subprocess work as a library, and instead move
Done
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@33
PS17, Line 33: private int queueSize;
> I meant, why bother defining constants for "q", "w", "m", and all the descr
I want to avoid adding inline comment /*blah=*/, also some of the variables are used multiple times (also in the tests).
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 18
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 07 Feb 2020 06:06:58 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has removed a vote on this change.
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 14
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#8).
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2971: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate over a stdin/stdout
pipe via protobuf-based protocol. It is useful in cases a Kudu process
(e.g. master) needs to talk to third-party libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal and causes the program to exit, e.g. I/O errors
when reading/writing to the pipe, and parsing malformed protobuf messages.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProcessorMain.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Subprocessor.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java
M java/settings.gradle
16 files changed, 1,355 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/8
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 8
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 16:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@104
PS15, Line 104: ith the underlying buffer size being the
: * maximum bytes of a message, the write is atomic. That i
> Does it mean the buffer size is always greater than the buffer in BufferedOutputStream
I guess you mean the buffer size of the BufferedOutputStream is always greater than (or at least equal to) the message size. Yes, it is the case as documented at https://gerrit.cloudera.org/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java#92 (the BufferedOutputStream is initiated with the maximum bytes of a message to ensure the underlying buffer can hold the entire message before flushing). Sorry I should be clear here as well, I will update the comment.
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@91
PS14, Line 91:
:
:
:
> Oh, I just meant that:
Ack
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java:
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@51
PS15, Line 51: private static Function<Throwable, Object> ERROR_HANDLER = (t) -> {
> Could be final?
As we have constructor assign errorHandler again(SubprocessExecutor(Function<Throwable, Object> errorHandler)), it is not possible to be final.
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@59
PS15, Line 59: SubprocessExecutor() {
: }
> Can be removed?
This is needed for non-test case in SubprocessExecutor.
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java:
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@81
PS15, Line 81: subprocessExecutor.run(args, protocolProcessor, /* timeoutMs= */1000);
: }
> Nit: move this to just after the declaration of LOG so that it's easier to
Done
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@104
PS15, Line 104: /**
> I think you can simplify this code by adding (expected = TimeoutException.c
Done
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@41
PS15, Line 41:
> reading and writing
Done
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@43
PS15, Line 43:
> Same comments about using an ExpectedException Rule and a RuleChain.
Done
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@53
PS15, Line 53:
: public static class PrintStreamOverload extends PrintStream {
: public PrintStreamOverload(OutputStream out) {
:
> Should add a comment explaining what's happening here: you're expanding the
Done
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 16
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 05 Feb 2020 22:13:39 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 2:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG@14
PS1, Line 14: reader thread th
> Could we just call them reader and writer then? Producer and consumer is a
Done
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG@15
PS1, Line 15: optional JSON encoding) from
> Can you elaborate on why this is useful? The main use case I know of is for
I think this is mainly a match up with the C++ side protobuf format support. Other than that, I can only think of better debuggability as message is human readable format.
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java@17
PS1, Line 17:
: package org.apache.kudu.subprocess;
:
: import org.apache.kudu.subprocess.Subprocess.EchoRequestPB;
: import org.apache.kudu.subprocess.Subprocess.EchoResponsePB;
:
: /**
: * Class that processes a EchoRequest and simply echoes the request
: * as a response.
: */
: public class EchoProtocolProcessor implements ProtocolProcessor<EchoRequestPB, EchoResponsePB> {
:
: @Override
: public EchoResponsePB response(EchoRequestPB request) {
: EchoResponsePB.Builder resBuilder = EchoResponsePB.newBuilder();
: resBuilder.setDat
> Delete
Done
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@42
PS1, Line 42: static final Strin
> Sorry if this is a newbie java question, but doesn't this mean that this is
Yeah, you are right! Sorry, my mistakes.
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@160
PS1, Line 160: BufferedInputStream in
> Why bother passing these around? Why not just make them members of MessageP
The given BufferedInputStream (or BufferedOutputStream) can be different each time the read/write method calls (especially for the test cases). And is not necessarily known to MessageProcessor at the initiation time.
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@95
PS1, Line 95: a.isEmp
> Should this be local to the MessageProducer rather than static?
Yeah, my mistake.
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@31
PS1, Line 31: Responds to a protobuf request messa
> nit: "Responds to a protobuf request message."
Done
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@38
PS1, Line 38: public class SubprocessCommandLineParser {
> Maybe a newbie java question, but why go through the trouble of building th
I think it is more specific in this way, as there are some configurations that can be local to different subprocess instances.
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProducerConsumer.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProducerConsumer.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProducerConsumer.java@131
PS1, Line 131: MessageConsumer consumer = new MessageConsumer(blockingQueue, messageProcessor, out);
: for (int i = 0; i < CONSUMER_TASK_NUM; i++) {
: CompletableFuture consumerFuture =
: CompletableFuture.runAsync(consumer, consumerService);
: consumerFuture.exceptionally(errorHandler);
: }
> It'd be nice if we had a test that stressed this with many input messages a
Noted, although I would prefer to move the stress test to a following commit.
http://gerrit.cloudera.org:8080/#/c/14329/1/src/kudu/subprocess/subprocess.proto
File src/kudu/subprocess/subprocess.proto:
http://gerrit.cloudera.org:8080/#/c/14329/1/src/kudu/subprocess/subprocess.proto@24
PS1, Line 24: message EchoRequestPB {
: required string data = 1;
: }
: message EchoResponsePB {
: required string data = 1;
: }
> Are java generics not expressive enough to allow us to just use EchoRequest
I guess so, but what is the intention behind? Moreover, with the current approach we can easily add more generic field (e.g request ID) in SubprocessRequestPB(or SubprocessResponsePB) that are required for all requests/responses, instead of having them in each request/response definition.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 14 Oct 2019 16:58:31 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG@15
PS1, Line 15: optional JSON encoding) from
> Why would we want to send JSON over the channel though?
Looked at your other patch, and I now get what you mean that you want to keep it aligned with the C++ support.
I still don't think we need it. IIUC, the point of doing that for the cluster CLI tool was to be able to do IPC from other languages where we might not have binary protobuf support (e.g. python).
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 15 Oct 2019 02:14:12 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 12:
(21 comments)
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java@48
PS12, Line 48: Executor
> nit: FWIW I think Subprocess for this was fine, considering that's what we
Maybe name it SubprocessExecutor? Subprocess is not available as the protobuf file has the name.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java@50
PS12, Line 50: private Function<Throwable, Object> errorHandler;
> Can this be static final too?
Done
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java@88
PS12, Line 88: // need to 1. ensure no partial message write due to auto flush
: // 2. PrintStream is sawlloing IOexception
> nit: so it's clear you're referring to the SubprocessOutputStream, maybe in
Done
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java@112
PS12, Line 112: else {
: readerFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
: CompletableFuture.allOf(writerFutures)
: .get(timeoutMs, TimeUnit.MILLISECONDS);
: }
> I'll probably get to this later, but why would we impose a time limit on th
Done
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java@29
PS12, Line 29: subprocess
> Remove?
Done
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@53
PS12, Line 53: The read is not atomic (partial read can happen if any
: * exceptions occur) and blocking (waits until the input is available).
> nit: maybe be more prescriptive here? E.g.
Done
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@103
PS12, Line 103: The write is atomic, that is if any exceptions occur, no partial
: * message should be written to the underlying output stream.
> Is it possible that we'll raise an exception in flush(), and then we're lef
By 'exception' you mean IOException, right? Yeah, I think so but that will cause the program to exit as well, so it wouldn't matter for the other end to receive partial message.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@113
PS12, Line 113: this
> nit: at first I thought we might want to synchronize on 'out' instead of 't
Done
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@91
PS12, Line 91: respondWithError
> nit: this doesn't actually send the response so it's not "responding" per s
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@95
PS4, Line 95:
> Done
Sorry, revisiting the comment here.. Actually in C++ side we are using int for max message bytes, https://github.com/apache/kudu/blob/master/src/kudu/subprocess/subprocess_protocol.cc#L45. Did I miss anything?
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java@25
PS12, Line 25: * The {@link SubprocessOutputStream} class is a wrapper around {@link PrintStream}
: * for explicitly re-throw <code>IOException</code> if any encountered in a
: * <code>PrintStream</code>. Because unlike other output streams, a <code>PrintStream</code>
: * never throws an <code>IOException</code>; instead, exceptional situations
: * merely set an internal flag that can be tested via the <code>checkError</code>
: * method.
> nit: looking around for other wrappers, maybe reword a bit?
Done
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java@47
PS12, Line 47: @Override
> nit: spacing
Done
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java@49
PS12, Line 49: out.flush();
> Seem like we might not need this?
Done
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@90
PS12, Line 90: Assert.assertTrue(false);
> Could we also display the error here?
You mean log the error if it ever happen? If so, done.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@156
PS12, Line 156: String[] args = {"-w", "1"};
> Is this important for this test?
Yeah, added the comment to explain why.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@194
PS12, Line 194: if needed.
> remove this
Done
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@195
PS12, Line 195: reader.interrupt();
> Rather than copying the guts of Executor in this test, could we instead plu
Yeah, I understand, but I chose not to do that as it would be weird to plumb the injection state to the Executor, it is not relevant to non-test cases at all and it is particular for MessageWriter. Since the focus is mainly on MessageWriter, I don't expect it will change much with the change of Executor. Let me know if you think otherwise or you think there is a good way to plumb the injection state.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@214
PS12, Line 214: CompletableFuture.anyOf(writerFutures)
: .get(1000, TimeUnit.MILLISECONDS);
> Can't we wait for all of them to finish? If there's nothing in the pipe, wo
The writer task never timeout and will block on getting new messages.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@39
PS12, Line 39: public class TestMessageIO {
> How difficult would it be to plumb a PrintStream down and then call setErro
So what is the purpose of such test in MessageIO? I thought we want that at TestEchoSubprocess to ensure IOException will cause the program to exit.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@58
PS12, Line 58: // Construct a message that it is not properly serialized with the message size
: // that causes maximum message size exceed exception.
: byte[] malformedMessage = "malformed message".getBytes(StandardCharsets.UTF_8);
> Can you be more explicit and explain the UTF-8 will expand this message to
Hmm, to avoid confusion, I update it to be with the size exceeds max explicitly.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@75
PS12, Line 75: byte[] body = "malformed message".getBytes(StandardCharsets.UTF_8);
> Does it make sense to add a test for what happens if there isn't enough dat
Can you be more specific about what "isn't enough data" mean in this case?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 12
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 03 Feb 2020 06:55:55 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#14).
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2971: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate over a stdin/stdout
pipe via protobuf-based protocol. It is useful in cases a Kudu process
(e.g. master) needs to talk to third-party libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal and causes the program to exit, e.g. I/O errors
when reading/writing to the pipe, and parsing malformed protobuf messages.
If encounter InterruptedException during placing/getting messages to/from
the queue, we consider it to be a signal to shutdown the task which
cause the program to exit as well.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoSubprocessMain.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
M java/settings.gradle
17 files changed, 1,446 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/14
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 14
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 17: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 17
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 06 Feb 2020 07:18:18 +0000
Gerrit-HasComments: No
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/14329/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/7//COMMIT_MSG@18
PS7, Line 18: from the queue, process them and write the responses to the standard
: output.
:
> I'm OK with it as long as the follow-up doesn't end up rewriting a large ch
Hmm, I don't see it will require rewriting a large chunk of the code in this patch. As far as I see, it will be mainly touching/breaking down the MessageWriter class. But let me know if you think otherwise.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@84
PS7, Line 84: // InterruptedException during the put, record the interruption
> What does it mean to "get interrupted"? I'll admit it's been a while since
My understanding of InterruptedException is to provide the blocking method a way to cancel the activity, and stop blocking early. But not all blocking methods are required to pay attention to interruption, e.g. blockingQueue.put() can get interrupted by another thread when the queue is full and it is waiting to place an element to the queue. I don't see a strong reason to cancel the task for such case here. However, at L59 we are constantly checking if interrupted status is set (may caused by your example "someone calls interrupt() on a single thread with the intent to have it die'"), then the while loop will break and the task will be cancelled. I also found this article helpful on understanding InterruptedException and how to handle it https://www.ibm.com/developerworks/library/j-jtp05236/.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@98
PS7, Line 98: }
> Won't the thread's interrupted status still be true, thus breaking out of t
Hmm, no, Thread.currentThread().isInterrupted() is actually checking the internal interrupted status instead of the one defined locally at L57.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/resources/log4j2.properties
File java/kudu-subprocess/src/main/resources/log4j2.properties:
PS7:
> Hmm, why do we provide a log4j2.properties file here at all? AFAICT in ever
I was thinking to use it as the default log config for the subprocess. Although we may need to add a file appender for production usage. Maybe I can move it so that it is only used for tests for now?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 8
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 14 Jan 2020 07:11:45 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14329/8/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/8/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@79
PS8, Line 79: if (injectInterrupt) {
why don't you simply call interrupt on the thread in the test?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 8
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 13 Jan 2020 12:17:51 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 20:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@33
PS17, Line 33: public class SubprocessConfiguration {
> But with the exception of the boolean, these are all pretty self explanator
Sounds good, updated.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 20
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 07 Feb 2020 20:13:52 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 4:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@95
PS4, Line 95: return conf.getInt(MAX_MESSAGE_BYTES, MAX_MESSAGE_BYTES_DEFAULT);
I think this should be long just in case. In the C++ side we use uint but all integers in Java are signed, so if we specify a max message size that is 2^31-1 < max_message_size < 2^32 it would be perfectly valid in the C++ side but would cause an overflow and a negative number here. Alternatively, we can make the C++ side signed.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@63
PS4, Line 63: try {
maybe we can use try with resources here and then join the threads at the end of the try block so that they won't be closed immediately. In this case we don't need to handle closing the streams in the consumer and producer.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@80
PS4, Line 80: MessageConsumer consumer = new MessageConsumer(blockingQueue, messageProcessor,
there should be a separate consumer instance for each thread.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@88
PS4, Line 88: producerService.shutdown();
don't think these are needed, shutdown only means the executor service won't accept new tasks. To actually interrupt them we would need to run shutdownNow() as these are all infinite loops that won't normally terminate. The finally block will be called right after the futures are submitted, so we shouldn't do this here, but in a shutdown hook instead so that if the subprocess receives a signal we can gracefully shut down.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java@78
PS4, Line 78: public static class NonParameterizedTest {
this should be in its own class, but as Andrew said, we probably don't need parameterized anymore the two classes can be merged.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 4
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 20 Nov 2019 15:36:35 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 16:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java:
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@51
PS15, Line 51: private static Function<Throwable, Object> ERROR_HANDLER = (t) -> {
> As we have constructor assign errorHandler again(SubprocessExecutor(Functio
Oh, I missed that usage. In that case, please move this initialization back to the constructor, remove the static keyword, and mark it as final. It'll just be a regular constructor-initialized class field then.
http://gerrit.cloudera.org:8080/#/c/14329/15/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@59
PS15, Line 59: SubprocessExecutor() {
: }
> This is needed for non-test case in SubprocessExecutor.
Right, I missed that there's another constructor. My bad.
http://gerrit.cloudera.org:8080/#/c/14329/16/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java:
http://gerrit.cloudera.org:8080/#/c/14329/16/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@47
PS16, Line 47: public RetryRule retryRule = new RetryRule();
Does this still need a @Rule annotation? Or not when paired with a RuleChain?
http://gerrit.cloudera.org:8080/#/c/14329/16/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/16/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@45
PS16, Line 45: public RetryRule retryRule = new RetryRule();
Same question.
http://gerrit.cloudera.org:8080/#/c/14329/16/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@116
PS16, Line 116: thrown.expect(IOException.class);
: thrown.expectMessage("exceeds maximum message size");
: messageIO.readBytes();
This works mid-test? I thought the call that throws an exception would have to be the last thing in the test. TIL.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 16
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 05 Feb 2020 22:23:47 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java@29
PS4, Line 29: @InterfaceAudience.Private
> Yes. Multiple jars makes more sense to me than a single JAR that has everyt
Makes sense. Updated.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 6
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 16 Dec 2019 18:28:42 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 17: Code-Review+1
(11 comments)
Overall looks good, just a few more nits.
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/build.gradle
File java/kudu-subprocess/build.gradle:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/build.gradle@46
PS17, Line 46: jar {
: manifest {
: attributes(
: 'Main-Class': 'org.apache.kudu.subprocess.EchoSubprocessMain'
: )
: }
: }
Is it expected that this builds kudu-subprocess.jar and not some echo-subprocess.jar?
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@83
PS17, Line 83: If fails to read
nit: "If it fails to read"
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@123
PS17, Line 123: a four bytes
nit: "a four-byte array", same below
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@88
PS17, Line 88: it the code higher
nit: "propagate it up the call stack"
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@43
PS17, Line 43: private static final Logger LOG = LoggerFactory.getLogger(MessageWriter.class);
Not sure if you're seeing this, but when I run the subprocess from a C++-managed process, I see this logged first:
log4j:WARN No appenders could be found for logger (org.apache.kudu.subprocess.MessageReader).
log4j:WARN No appenders could be found for logger (org.apache.kudu.subprocess.MessageWriter).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
http://gerrit.cloudera.org:8080/#/c/14329/13/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/13/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@42
PS13, Line 42: X_WR
nit: add a space
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@33
PS17, Line 33: private static final String QUEUE_SIZE_SHORT_OPTION = "q";
What's the point of defining all these variables up front instead of in-lining them where they're used? This isn't a pattern we really see in our C++ tooling; why here? Same with other files and log error messages.
In general, "magic numbers" deserve constants; but these don't seem magical at all, and having constants means jumping back and forth between where they're used and where they're defined to understand things.
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@34
PS17, Line 34: queuesize
nit: camelCase for this?
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@36
PS17, Line 36: Size of the message queue for subprocess"
nit: maybe "Maximum number of messages held by the message queue"? so it's clear the units here are messages and not bytes or something.
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@95
PS17, Line 95: final Function<Throwable, Object> noErrors = e -> {
: LOG.error(String.format("Unexpected error: %s", e.getMessage()));
: Assert.assertTrue(false);
: return null;
: };
nit: maybe declare this to be static as NO_ERRORS and reuse it elsewhere. Same with HAS_ERROR?
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@180
PS17, Line 180: final boolean injectInterrupt = true;
nit: why not inline this and annotate with a comment? Same above?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 17
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 06 Feb 2020 18:59:09 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 4:
(19 comments)
Didn't do a full review of the tests. Overall looking much better!
http://gerrit.cloudera.org:8080/#/c/14329/4//COMMIT_MSG
Commit Message:
PS4:
Could you describe the expectations w.r.t error handling? What kind of "errors" are OK? What errors return a response? What errors are fatal? What happens when there is a fatal error? How do interrupts tie into this mental model for error handling?
http://gerrit.cloudera.org:8080/#/c/14329/4//COMMIT_MSG@22
PS4, Line 22: simply extend the 'ProtocolProcessor'
Seems you also have do update the SubprocessConfiguration
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java@29
PS4, Line 29: @InterfaceAudience.Private
I like the use of generics here. Could we extend it even further so that we only have to define a single EchoProcessorMain.java or RangerProcessorMain.java or AtlasProcessorMain.java when we add new processes, rather than adding an EchoProtocolProcessor.java _and_ updating SubprocessConfiguration.java?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java@32
PS4, Line 32: private static String ECHO_MESSAGE = "echo";
nit: final?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@35
PS4, Line 35: /**
: * Message consumer class that retrieve a message from the queue at a time.
: * Then process the message and write the response to the underlying output
: * stream.
: */
nit: maybe note the concurrency expectations of this method, e.g. that we can expect to be running multiple MessageConsumers with a single, shared BufferedOutputStream
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@47
PS4, Line 47: private AtomicInteger retries;
Could this be local to each call to run()? If so, could it be non-atomic?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@115
PS4, Line 115: // Restore the interrupted status to the code higher up
: // on the call stack.
: if (interrupted) {
: Thread.currentThread().interrupt();
: }
This never gets reset to false, even if we were eventually table to take from the queue. What purposes does throwing an exception serve? Don't we only get here if we're throwing an exception anyway?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@39
PS4, Line 39: class MessageProcessor {
Why does this need to be a class? Couldn't it be a set of utility functions that reference Subprocess.MAX_MESSAGE_BYTES_DEFAULT?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@40
PS4, Line 40:
: public enum SerializationMode {
: /**
: * Each message is serialized as a four byte big-endian size followed by
: * the protobuf-encoded message itself.
: */
: PB
: }
Remove this?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@50
PS4, Line 50: private static final int OFFSET = 0;
nit: probably don't need to define a constant for this
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@75
PS4, Line 75: return data;
nit: don't need the local variable -- just:
return new String()
?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@93
PS4, Line 93: void writeMessage(BufferedOutputStream out,
: Message message) throws IOException {
: writeBinaryMessage(out, message);
: }
Why have this when writeBinaryMessage already exists?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@113
PS4, Line 113: * Read a protobuf message in {@link SerializationMode#PB} mode from
: * the given buffered input stream.
nit: maybe note that this isn't threadsafe and that we expect there to be a single reader of 'in'
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@126
PS4, Line 126: Preconditions.checkState(read == Integer.BYTES,
: "unable to receive message size");
: int size = bytesToInt(sizeBytes);
: Preconditions.checkState(size <= maxMessageBytes,
: "exceeds maximum message size");
Can we also include 'read' and 'size' in the error messages to facilitate debugging?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@39
PS4, Line 39: should not be called concurrently unless handled by
: * the caller.
I'm finding this a bit confusing: under what scenario can run() be called concurrently?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@98
PS4, Line 98: continue;
Why aren't we responding with an error here?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@37
PS4, Line 37: SubprocessRequestPB
SubprocessResponsePB
Same below
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@64
PS4, Line 64: abstract Class<Request> getRequestClazz();
Will this always be:
Class<Request> getRequestClazz() {
return Request.class;
}
? If so, could we implement it here instead of in the subclasses? Alternatively, could we inline the call so it's something like:
getResponse(request.getRequest.unpack(Request.class))
?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java@42
PS4, Line 42: * Parameterized the message processor test to run in both serialization
: * mode and message request type.
There's only one serialization mode now.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 4
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 19 Nov 2019 02:36:50 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14329/1/src/kudu/subprocess/subprocess.proto
File src/kudu/subprocess/subprocess.proto:
http://gerrit.cloudera.org:8080/#/c/14329/1/src/kudu/subprocess/subprocess.proto@24
PS1, Line 24: message EchoRequestPB {
: required string data = 1;
: }
: message EchoResponsePB {
: required string data = 1;
: }
> message EchoRequestPB {
Ah I see this is based off of the work in tools/tool.proto.
Adar had this to say there:
// Because the control shell communicates via pipe and not krpc, we can't make
// use of service dispatch and must instead multiplex all command requests and
// responses via ControlShellRequestPB and ControlShellResponsePB respectively.
This is saying that the reason it is written with oneof instead of the header approach is that the cluster CLI needs to be able to perform any of the Create/Delete/Start/etcCluster commands. That isn't the case here -- I'd guess each Java subprocess only cares about processing one kind of request, similar to krpc. If so, I would be more in favor of using headers instead of multiplexing.
I left feedback on the other patch that maybe we should reuse less of the cluster CLI.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 15 Oct 2019 01:55:32 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 20: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 20
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 07 Feb 2020 20:35:37 +0000
Gerrit-HasComments: No
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 17:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/build.gradle
File java/kudu-subprocess/build.gradle:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/build.gradle@46
PS17, Line 46: jar {
: manifest {
: attributes(
: 'Main-Class': 'org.apache.kudu.subprocess.EchoSubprocessMain'
: )
: }
: }
> Is it expected that this builds kudu-subprocess.jar and not some echo-subpr
Yeah, EchoSubprocessMain is the default protocol supported for kudu-subprocess. But if it sounds confusing, I can rename it DefaultSubprocessMain? Which on the other hand lost the track of what protocol it is supporting.
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@43
PS17, Line 43: private static final Logger LOG = LoggerFactory.getLogger(MessageWriter.class);
> Not sure if you're seeing this, but when I run the subprocess from a C++-ma
Hmm, I think I saw this as well with C++ patch, it seems somehow the logger isn't picked up. Haven't figured it out why, I am planning to address it with the follow up patch (with C++).
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 17
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 06 Feb 2020 20:12:15 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 5:
(14 comments)
http://gerrit.cloudera.org:8080/#/c/14329/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/5//COMMIT_MSG@22
PS5, Line 22: IOExcpetion
nit: IOException
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java@29
PS4, Line 29: @InterfaceAudience.Private
> I don't quite understand what will your proposal look like. Do you mean to
Yes. Multiple jars makes more sense to me than a single JAR that has everything (Atlas, Ranger, etc.) in it, especially because it means that we can update/deprecated/etc them more independently.
The idea would be that RangerProcessorMain (or similar) would define all of these overrides, as well as the members/functions in ProtocolProcessor. That way there would be a single file that defines everything we need for the Echo service, Ranger service, Atlas service, etc. And we wouldn't have to rely on passing configs to pick the service we want to run since each service would run in its own JAR.
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@77
PS5, Line 77: binary mode
nit: "binary mode" doesn't exist in the context of the java subprocess anymore, so maybe remove this? Same below.
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@85
PS5, Line 85: * @throws IllegalArgumentException if the serialization mode is invalid
: * @throws IllegalStateException if the message is malformed
nit: remove this
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@88
PS5, Line 88: String readMessage() throws IOException {
nit: Maybe we should add preconditions here that 'in' is set.
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@88
PS5, Line 88: readMessage
nit: how about calling this readBytes() or somesuch?
I know we're technically reading profobuf messages, but at this point in the process, all we know is that we're dealing with bytes, and so calling it readBytes() makes it more obvious that we haven't done a whole lot of PB parsing yet.
Same with parseMessage. Maybe call it parseBytes()? or bytesToMessage() or something similar?
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@117
PS5, Line 117: * @throws IllegalArgumentException if the serialization mode is invalid
: * @throws InvalidProtocolBufferException if there are any unknown types
: * in the message
nit: remove this
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@122
PS5, Line 122: void writeMessage(Message message) throws IOException {
nit: Maybe we should add preconditions here that 'out' is set.
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@131
PS5, Line 131: protobuf message with the given parser and builder.
nit: we're not actually parsing a protobuf message -- we're parsing some bytes and creating a protobuf message using the given parser, right?
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@33
PS5, Line 33: read
nit: reads
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@37
PS5, Line 37: write
nit: writes
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java:
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@67
PS5, Line 67: Function<Throwable, Object> errorHandler = (t) -> {
: System.exit(1);
: return null;
: };
nit: Could this be static?
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@78
PS5, Line 78: producer
nit: we've shifted over to calling these 'readers' and 'writers'. Could you update variable names, comments, and tests accordingly?
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java:
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@69
PS5, Line 69: MessageProcessor messageProcessor = new MessageProcessor(
: SubprocessConfiguration.MAX_MESSAGE_BYTES_DEFAULT);
Do we have to have a different constructor for this? Couldn't we just pass in byteOutputStream? Same below?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 5
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 09 Dec 2019 22:59:58 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#20).
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2971: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate over a stdin/stdout
pipe via protobuf-based protocol. It is useful in cases a Kudu process
(e.g. master) needs to talk to third-party libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal and causes the program to exit, e.g. I/O errors
when reading/writing to the pipe, and parsing malformed protobuf messages.
If encounter InterruptedException during placing/getting messages to/from
the queue, we consider it to be a signal to shutdown the task which
cause the program to exit as well.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess-echo/build.gradle
A java/kudu-subprocess-echo/src/main/java/org/apache/kudu/subprocess/echo/EchoProtocolHandler.java
A java/kudu-subprocess-echo/src/main/java/org/apache/kudu/subprocess/echo/EchoSubprocessMain.java
A java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
M java/settings.gradle
17 files changed, 1,395 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/20
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 20
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#18).
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2971: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate over a stdin/stdout
pipe via protobuf-based protocol. It is useful in cases a Kudu process
(e.g. master) needs to talk to third-party libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal and causes the program to exit, e.g. I/O errors
when reading/writing to the pipe, and parsing malformed protobuf messages.
If encounter InterruptedException during placing/getting messages to/from
the queue, we consider it to be a signal to shutdown the task which
cause the program to exit as well.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoSubprocessMain.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
M java/settings.gradle
16 files changed, 1,367 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/18
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 18
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#13).
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2971: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate over a stdin/stdout
pipe via protobuf-based protocol. It is useful in cases a Kudu process
(e.g. master) needs to talk to third-party libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal and causes the program to exit, e.g. I/O errors
when reading/writing to the pipe, and parsing malformed protobuf messages.
If encounter InterruptedException during placing/getting messages to/from
the queue, we consider it to be a signal to shutdown the task which
cause the program to exit as well.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoSubprocessMain.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
M java/settings.gradle
17 files changed, 1,408 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/13
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 13
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#16).
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2971: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate over a stdin/stdout
pipe via protobuf-based protocol. It is useful in cases a Kudu process
(e.g. master) needs to talk to third-party libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal and causes the program to exit, e.g. I/O errors
when reading/writing to the pipe, and parsing malformed protobuf messages.
If encounter InterruptedException during placing/getting messages to/from
the queue, we consider it to be a signal to shutdown the task which
cause the program to exit as well.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoSubprocessMain.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
M java/settings.gradle
16 files changed, 1,388 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/16
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 16
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#2).
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2791: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate with other process
via protobuf-based protocol over standard input/output. It is useful
in cases a Kudu process (e.g. master) needs to talk to third-party
libraries written in Java.
This tool has: 1) a single reader thread that continuously reads
protobuf-based messages (with optional JSON encoding) from the standard
input and puts the messages to a FIFO blocking queue;
2) multiple writer threads that continuously retrieve the messages from
the queue, process them and write the responses to the standard output.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface to specify how to handle the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
M java/gradle/dependencies.gradle
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLine.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessRuntimeException.java
A java/kudu-subprocess/src/main/resources/log4j.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProducerConsumer.java
M java/settings.gradle
17 files changed, 1,387 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/2
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/build.gradle
File java/kudu-subprocess/build.gradle:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/build.gradle@25
PS6, Line 25: compile libs.hadoopCommon
Is this included just for Configuration.Java? There should be something more lightweight that can be used.
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/build.gradle@56
PS6, Line 56: configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }
The shadow plugin, included above, should handle the shading of this jar for you. Is this required?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 6
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 16 Dec 2019 22:08:15 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 5:
(24 comments)
http://gerrit.cloudera.org:8080/#/c/14329/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/5//COMMIT_MSG@22
PS5, Line 22: IOExcpetion
> nit: IOException
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@70
PS4, Line 70:
> hm... there's no way to gracefully shut down in this case though. If we don
As mentioned earlier, it is advised to 'Restore the interruption status so that code higher up on the call stack can deal with it' in Java Concurrency in Practice' Chapter 7 (p143). Even though we don't consider InterruptedException from the BlockingQueue fatal, we still want the caller to know about it when fatal exception happen.
However, as you suggested to register a shutdown hook for orderly shutdown, we need to handle InterruptedException from ExecturorService.shutdownNow(), so updated it with your suggestion.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@148
PS4, Line 148: */
> even if BufferedOutputStream synchronizes, write and flush, I think it shou
I updated as you suggested to synchronized on out.
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@77
PS5, Line 77: binary mode
> nit: "binary mode" doesn't exist in the context of the java subprocess anym
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@85
PS5, Line 85: * @throws IllegalArgumentException if the serialization mode is invalid
: * @throws IllegalStateException if the message is malformed
> nit: remove this
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@88
PS5, Line 88: readMessage
> nit: how about calling this readBytes() or somesuch?
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@88
PS5, Line 88: String readMessage() throws IOException {
> nit: Maybe we should add preconditions here that 'in' is set.
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@89
PS5, Line 89: if (in.available() <= 0) {
> can this be negative? anyway, we should return "" instead of new String() s
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@117
PS5, Line 117: * @throws IllegalArgumentException if the serialization mode is invalid
: * @throws InvalidProtocolBufferException if there are any unknown types
: * in the message
> nit: remove this
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@122
PS5, Line 122: void writeMessage(Message message) throws IOException {
> nit: Maybe we should add preconditions here that 'out' is set.
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@125
PS5, Line 125: out.write(Bytes.concat(size, body));
: // Always do a flush after write to ensure no partial message is written.
: out.flush();
> these two statements should be synchronized on out.
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@131
PS5, Line 131: protobuf message with the given parser and builder.
> nit: we're not actually parsing a protobuf message -- we're parsing some by
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@33
PS5, Line 33: read
> nit: reads
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@60
PS5, Line 60: true
> this should be while (!Thread.currentThread().isInterrupted()). Right now t
Updated as calling ExecutorService.shutdownNow() may interrupt the thread.
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@94
PS5, Line 94: LOG.debug("Message: {} has been put on the queue", data);
> this should be in an if (LOG.isDebugEnabled()) block so we don't format a s
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@96
PS5, Line 96: } catch (InterruptedException e) {
> why are we retrying when the thread is interrupted? Shouldn't we simply int
I don't see it is necessary to cancel the activity that puts the message in the queue if it get blocked and interrupted immediately, it may success with following retry. Moreover, even if 'blockingQueue.put' fail after several retries, we don't want a single message process failure to cause the program to fail. Or I am missing anything?
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@37
PS5, Line 37: write
> nit: writes
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@63
PS5, Line 63: while (true) {
> this should be while (!Thread.currentThread().isInterrupted()) as well, sim
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@69
PS5, Line 69: LOG.debug("Message: {} has been taken from the queue", data);
> if (LOG.isDebugEnabled())
Done
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@88
PS4, Line 88: consumerFuture.exceptionally(errorHandler);
> yeah we can skip shutdowns here, but maybe add them in a shutdown hook?
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java:
PS5:
> we should register a shutdown hook that shuts down the threadpools (shutdow
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@67
PS5, Line 67: Function<Throwable, Object> errorHandler = (t) -> {
: System.exit(1);
: return null;
: };
> nit: Could this be static?
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@78
PS5, Line 78: producer
> nit: we've shifted over to calling these 'readers' and 'writers'. Could you
Done
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java:
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@69
PS5, Line 69: MessageProcessor messageProcessor = new MessageProcessor(
: SubprocessConfiguration.MAX_MESSAGE_BYTES_DEFAULT);
> Do we have to have a different constructor for this? Couldn't we just pass
Done
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 5
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 16 Dec 2019 07:15:27 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/14329/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/7//COMMIT_MSG@18
PS7, Line 18: from the queue, process them and write the responses to the standard
: output.
:
> Makes sense. I am thinking to address it in a follow up patch as this one i
I'm OK with it as long as the follow-up doesn't end up rewriting a large chunk of the code in this patch. If it does, I'd rather the follow-up be part of this patch so as to minimize overall churn and simplify the review.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@84
PS7, Line 84: // InterruptedException during the put, record the interruption
> In this particular case, I don't think we would want to cancel the read/wri
What does it mean to "get interrupted"? I'll admit it's been a while since I wrote systems-level Java code, but my understanding is that it means, for example, receiving a UNIX signal (a la Ctrl-C). That's not a recoverable condition; that's a condition wherein the process is expected to die. Another example: someone calls interrupt() on a single thread with the intent to have it die.
Are there situations where InterruptedException is thrown for run-of-the-mill normal events that a process is expected to actually handle? Or is it only used in cases where we need to shutdown but we want to ensure a more orderly shutdown?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@98
PS7, Line 98: }
> Hmm, not quite following your question, here we retry put() for certain tim
Won't the thread's interrupted status still be true, thus breaking out of the while loop on :59?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/resources/log4j2.properties
File java/kudu-subprocess/src/main/resources/log4j2.properties:
PS7:
> I think that is to specify the log level for org.apache.kudu package specif
Hmm, why do we provide a log4j2.properties file here at all? AFAICT in every other instance it's only for tests.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 8
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 14 Jan 2020 06:13:22 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 18:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@83
PS17, Line 83: read the specifie
> nit: "If it fails to read"
Done
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@123
PS17, Line 123: ta a four-byt
> nit: "a four-byte array", same below
Done
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@88
PS17, Line 88: it up the call sta
> nit: "propagate it up the call stack"
Done
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@33
PS17, Line 33: private int queueSize;
> What's the point of defining all these variables up front instead of in-lin
Done
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@34
PS17, Line 34:
> nit: camelCase for this?
Done
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@36
PS17, Line 36: AULT = 3;
> nit: maybe "Maximum number of messages held by the message queue"? so it's
Done
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java:
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@95
PS17, Line 95: */
: @Test(expected = TimeoutException.class)
: public void testBasicMsg() throws Exception {
: final String message = "data";
: fi
> nit: maybe declare this to be static as NO_ERRORS and reuse it elsewhere. S
Done
http://gerrit.cloudera.org:8080/#/c/14329/17/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@180
PS17, Line 180:
> nit: why not inline this and annotate with a comment? Same above?
Done
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 18
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 06 Feb 2020 21:33:24 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14329/8/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/8/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@79
PS8, Line 79: if (injectInterrupt) {
> why don't you simply call interrupt on the thread in the test?
Here we want to test InterruptionException thrown for blockingQueue.put(data) specifically. If we interrupt the test thread the whole task fail instead.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 8
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 13 Jan 2020 19:39:35 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 14:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@195
PS12, Line 195: Assert.assertTrue
> Sure, I agree that it doesn't feel great to intermingle test-only code with
Hmm, it is actually hard to not include/use CompletableFuture(no Executor) in this case as we want to verify the error handling of Executor as well. So after pondering a bit, I took a step back to plumb the inject state into Executor. I try to make the code looks clean, but let me know how you think. Thanks!
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@210
PS12, Line 210:
:
> I don't really follow this comment. Mind rewriting it? Also probably don't
Removed.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@214
PS12, Line 214:
:
> Could we move the readerFuture.get() call out of the try/catch?
Removed.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@39
PS12, Line 39:
> Having explicit tests for each component of a large system in isolation see
Discussed offline, now I got Andrew's comment is about test SubprocessOutputStream. Updated to add such case.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@75
PS12, Line 75: * catch errors thrown from underlying <code>PrintStream</code> and r
> Say we send over the following message:
Oh, this test case is for such scenario (mismatched size and body). I will update it to be more specific.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 14
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 04 Feb 2020 07:47:14 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 8:
(37 comments)
http://gerrit.cloudera.org:8080/#/c/14329/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/7//COMMIT_MSG@7
PS7, Line 7: KUDU-2971
> KUDU-2971
Done
http://gerrit.cloudera.org:8080/#/c/14329/7//COMMIT_MSG@9
PS7, Line 9: a std
> Should clarify here that because communication takes place over a stdin/std
Done
http://gerrit.cloudera.org:8080/#/c/14329/7//COMMIT_MSG@18
PS7, Line 18: from the queue, process them and write the responses to the standard
: output.
:
> Curious why you didn't go with writing the messages to another blocking que
Makes sense. I am thinking to address it in a follow up patch as this one is big enough?
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/BasicSubprocessor.java@96
PS6, Line 96:
:
:
> Ah, I was asking how this interacts, if at all, with ERROR_HANDLER, but bas
Ah, that is not only for output stream, updated. Thanks!
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@44
PS6, Line 44:
: MessageProcessor(long maxMessageBytes,
: BufferedInputStream in,
: BufferedOutputStream out) {
: this.maxMessageBytes = maxMessageBytes;
: this.in = in;
: this.out = out;
: }
:
: /**
: * Read a protobuf message, if any, from the underlying buffered input
: * stream. The read is not atomic (partial read can happen if any
: * exceptions occur) and blocking (waits until the input is available).
: *
: * @return the message in a byte array.
: * @throws EOFException if the end of the stream has been reached
: * @throws IOException if this input stream has been closed, an I/O
:
> I just requested constructors on the latest patch. I think one constructor
Done
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@92
PS6, Line 92: private void doRead(byte bytes[], int size) throws EOFExcept
> IllegalStateException is a RuntimeException which is an "Unchecked" excepti
Ack
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@49
PS7, Line 49: this.in = in;
> Can this be in the constructor instead? Mutable state like this can get tri
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@72
PS7, Line 72: String.format("message size (%d) exceeds maximum message size (%d)",
: size, maxMessageBytes));
: }
> Why do we do this? Don't we want to block waiting for additional input if n
Makes sense, updated.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@83
PS7, Line 83: * input stream into the specified byte array, starting at the offset
: * <code>0</code>. If fail to r
> Nit: reformat using String.format
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@90
PS7, Line 90: * error occurs, or fail to read the specified size
> Maybe include how much was actually read?
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@91
PS7, Line 91: */
> Why are we converting this into a UTF-8 String? It's a serialized PB messag
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@122
PS7, Line 122:
> Why is this a String? Shouldn't it be a byte array?
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@133
PS7, Line 133: }
:
: /**
> Nit: could combine:
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@145
PS7, Line 145:
: /**
: * Convert a 32-bit integer to a four bytes array in big endian order.
: * @param value a 32-bit
> Nit: could combine:
Done
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@98
PS6, Line 98: }
> I'm asking whether that's what we're doing. 'finally' blocks are run uncond
yeah, this still interrupts the caller if it cares about InterruptedException. This is to follow regular guideline of how to handling InterruptedException. However, in this case the caller(CompletableFuture.runAsync in Subprocessor.java) does not check for the interrupt flag based on my understanding. So it doesn't matter much to propagate or not.
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@98
PS6, Line 98: }
> Is there a test case around this retry/finally handling that verifies/show
Added one.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@44
PS7, Line 44: private BlockingQueue<byte[]> blockingQueue;
> Should be final too.
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@60
PS7, Line 60:
: while (!Thread.currentThread().isInterrupted()) {
: // Read the message from the standard input. If f
> How is this safe? AFAICT, in most cases that readBytes() throws IllegalStat
Makes sense, updated.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@84
PS7, Line 84: // InterruptedException during the put, record the interruption
> Why do we retry if we've been interrupted? Isn't interruption a sign that t
In this particular case, I don't think we would want to cancel the read/write task and exit the subprocess program due to InterruptedException thrown from the blocking queue (e.g cases like waiting on put the message to the queue and get interrupted). Therefore I chose to retry. But maybe you see there is a strong reason we should exit in such cases?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@98
PS7, Line 98: }
> IIUC, interruption means we retry the put() but then exit the main loop. Wh
Hmm, not quite following your question, here we retry put() for certain times and continue (does not exit the main loop) either the retry succeeds or fails. And the retry is to try not lose a received message if possible.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@60
PS7, Line 60: boolean interrupted = false;
> Same questions about interruption as for MessageReader.
Ack
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@130
PS7, Line 130: * and should be propagated to the code higher up on the call stack.
> I don't understand why we retry here at all, or what limits we're talking a
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java@64
PS7, Line 64:
> Nit: the 'clazz' variant is only necessary for variable names, where 'class
Done
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@69
PS6, Line 69: QUEUE_SIZE_HAS_ARG,
> I think I agree with Andrew that we could start with command line arguments
I see, updated to remove the config file.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@35
PS7, Line 35: String maxWriterThreads,
> This number feels really high to me, especially as a default. Is there any
Hmm, I don't have any data to back it up yet. Maybe to start it with 3 as default for now?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@60
PS7, Line 60:
> Nit: this (and associated fields) should be plural i.e. maxWriterThreads.
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Subprocessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Subprocessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Subprocessor.java@69
PS7, Line 69:
> Nit: add /* fair= */ here.
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Subprocessor.java@79
PS7, Line 79: BufferedOutputStream out = new BufferedOutputStream(System.out)) {
> Do we need to need to join on all of these futures? Via something like Comp
Yeah, right, with try-with-resource, we need to add join on the futures. Updated. Thanks!
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/resources/log4j2.properties
File java/kudu-subprocess/src/main/resources/log4j2.properties:
PS7:
> Every other log4j2.properties file also has this section:
I think that is to specify the log level for org.apache.kudu package specifically. Since debug level seems too low as default so skip it here. Let me know if you think otherwise.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@33
PS7, Line 33:
> Could you rename this to make it more clear that it's a test fixture? Maybe
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@38
PS7, Line 38:
> Nit: for new files, try to use indicative tense ("Constructs a ...") vs. an
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@45
PS7, Line 45:
> Nit: maybe 'create' or 'build' would be more precise?
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@46
PS7, Line 46:
> I don't understand how this method is going to be extensible for other mess
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@68
PS7, Line 68:
> Nit: maybe just serializeMessage?
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@78
PS7, Line 78:
> Sort of expected this to be more symmetric with getSerializedMessage. Maybe
Sorry for the confusion, updated it to be more symmetric and hope it is clear now.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTest.java@86
PS7, Line 86:
> Nit: and deserializeMessage?
Done
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java:
PS7:
> New tests should include the RetryRule so that they properly retry on failu
Done
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 8
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 13 Jan 2020 07:03:07 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@72
PS7, Line 72: String.format("message size (%d) exceeds maximum message size (%d)",
: size, maxMessageBytes));
: }
> Makes sense, updated.
As InputStream doesn't throw InterruptedException when it's interrupted (it will just block continuously), I would suggest checking availability and if there's nothing on the stream, sleep a bit and check again. The problem with this approach is that that if we choose a low value, we'll be busy waiting, but if it's too large, then we introduce latency. Maybe make the sleep configurable? What do you think?
Another (hacky) approach would be to send SIGINT then write a GOODBYE message to the stream to unblock?
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@84
PS7, Line 84: // InterruptedException during the put, record the interruption
> My understanding of InterruptedException is to provide the blocking method
If the thread is interrupted, we should terminate. If the blocking queue is full, there's no guarantee it will ever be consumed and that the put can succeed and we still want to terminate.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@98
PS7, Line 98: }
> Hmm, no, Thread.currentThread().isInterrupted() is actually checking the in
we still only interrupt the thread in finally which is basically never reached unless we receive an unchecked exception or when we throw KuduSubprocessException. We should simply Thread.currentThread().interrupt() here and remove the finally block.
http://gerrit.cloudera.org:8080/#/c/14329/8/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/8/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@79
PS8, Line 79: if (injectInterrupt) {
> Here we want to test InterruptionException thrown for blockingQueue.put(dat
I'm not sure I understand, the thread would be blocking on either messageProcessor.readBytes() or blockingQueue.put() anyway. If you use a mock blocking queue or a full real blocking queue in the test and you have something to read in the input stream, then interrupt the thread, blockingQueue will throw InterruptedException.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 8
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 14 Jan 2020 17:55:42 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 4:
(15 comments)
haven't finished reviewing yet, but I'm leaving a few comments.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@47
PS4, Line 47: private AtomicInteger retries;
> Could this be local to each call to run()? If so, could it be non-atomic?
yeah it should be local non-atomic so we retry 3 times for each message instead of 3 times total - and then not do anything for the remaining lifetime of the process
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@67
PS4, Line 67: public void run() {
we should probably split this method into 3 methods: take from queue, process message and build response, then write to the output.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@69
PS4, Line 69: try {
you can use try with resources here. If you do `try (BufferedOutputStream os = out)` it will automatically close the stream and you don't need to handle it in finally. Coupled with my suggestion in L64 you don't even need the finally block.
Maybe we shouldn't even close the streams here, but do a try with resources where we instantiate the consumers and producers as the streams could be shared between threads in which case we shouldn't close them.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@70
PS4, Line 70: true
this should be `while (!Thread.currentThread().isInterrupted())` instead. This doesn't change the interrupted flag either so there's no need for a separate interrupted flag in this scope.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@77
PS4, Line 77: } catch (InterruptedException e) {
We shouldn't catch this here, we don't want to retry anything if the thread is interrupted. The point is to gracefully interrupt the process. In this case what we should is jump to a catch of the outer try block and log that we were interrupted and let the process exit.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@94
PS4, Line 94: response = respondWithError(AppStatusPB.ErrorCode.ILLEGAL_STATE, responseBuilder);
I think we shouldn't respond here as we don't have a sequence ID so the server can't connect it back to a request anyway. We should retry after a timeout on the server side instead.
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@160
PS1, Line 160:
> The given BufferedInputStream (or BufferedOutputStream) can be different ea
Wouldn't it be still better to instantiate a separate MessageProcessor for each IO stream?
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@75
PS4, Line 75: return data;
> nit: don't need the local variable -- just:
or just `return "";`
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@148
PS4, Line 148: private void writeBinaryMessage(BufferedOutputStream out,
this should be synchronized. Alternatively (probably a better approach) would be if we had two queues - one for input and one for output.
We could have a MessageReader that reads from the InputStream and puts the messages on the input queue, the MessageProcessor would process the messages and put the response to the output queue and an MessageWriter would take the message from the output queue and write to the output stream. This way we can have a thread pool of MessageProcessors doing the actual work and a single MessageReader and MessageWriter that would take care of IO on the pipe.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java:
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@39
PS4, Line 39: should not be called concurrently unless handled by
: * the caller.
> I'm finding this a bit confusing: under what scenario can run() be called c
you could instantiate a bunch of MessageProducers in a thread pool.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@70
PS4, Line 70: try {
same as MessageConsumer
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@71
PS4, Line 71: while (true) {
same as MessageConsumer
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@88
PS4, Line 88: messageProcessor.writeMessage(out, response);
same as MessageConsumer, we don't have a sequence ID so we should just log the exception and drop the request.
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@100
PS4, Line 100: while (retries.getAndDecrement() > 0) {
same as MessageConsumer
http://gerrit.cloudera.org:8080/#/c/14329/4/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@106
PS4, Line 106: interrupted = true;
we shouldn't retry after an interruption, as in this case the expected behavior is to gracefully interrupt. If there's an InterruptedException thrown by blockingQueue.put(), we should do `Thread.currentThread().interrupt()` and then break out the retry loop.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 4
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 19 Nov 2019 16:04:09 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#17).
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2971: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate over a stdin/stdout
pipe via protobuf-based protocol. It is useful in cases a Kudu process
(e.g. master) needs to talk to third-party libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal and causes the program to exit, e.g. I/O errors
when reading/writing to the pipe, and parsing malformed protobuf messages.
If encounter InterruptedException during placing/getting messages to/from
the queue, we consider it to be a signal to shutdown the task which
cause the program to exit as well.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoSubprocessMain.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
M java/settings.gradle
16 files changed, 1,394 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/17
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 17
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2971: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate over a stdin/stdout
pipe via protobuf-based protocol. It is useful in cases a Kudu process
(e.g. master) needs to talk to third-party libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal and causes the program to exit, e.g. I/O errors
when reading/writing to the pipe, and parsing malformed protobuf messages.
If encounter InterruptedException during placing/getting messages to/from
the queue, we consider it to be a signal to shutdown the task which
cause the program to exit as well.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Reviewed-on: http://gerrit.cloudera.org:8080/14329
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Reviewed-by: Andrew Wong <aw...@cloudera.com>
Tested-by: Kudu Jenkins
---
A java/kudu-subprocess-echo/build.gradle
A java/kudu-subprocess-echo/src/main/java/org/apache/kudu/subprocess/echo/EchoProtocolHandler.java
A java/kudu-subprocess-echo/src/main/java/org/apache/kudu/subprocess/echo/EchoSubprocessMain.java
A java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
M java/settings.gradle
17 files changed, 1,395 insertions(+), 0 deletions(-)
Approvals:
Adar Dembo: Looks good to me, but someone else must approve
Andrew Wong: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 21
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 7:
(6 comments)
I did a partial review last night and will post my comments. Some may overlap with reviews since.
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@44
PS6, Line 44:
: MessageProcessor(long maxMessageBytes) {
: this.maxMessageBytes = maxMessageBytes;
: }
:
: void setInputStream(BufferedInputStream in) {
: Preconditions.checkNotNull(in);
: this.in = in;
: }
:
: void setOutputStream(BufferedOutputStream out) {
: Preconditions.checkNotNull(out);
: this.out = out;
: }
:
: /**
: * Read a protobuf message, if any, from the underlying buffered input
:
> Done
I just requested constructors on the latest patch. I think one constructor is all that is needed right? Just the one that take all the fields? Sure a couple MessageTest test only need one, but the other can just be null then.
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@92
PS6, Line 92: }
> Ah, didn't realize Preconditions meant we didn't have to annotate with 'thr
IllegalStateException is a RuntimeException which is an "Unchecked" exception in Java and therefore isn't checked at compile-time and doesn't need to be exposed via the throws keyword.
All Unchecked exceptions are direct sub classes of RuntimeException class.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@49
PS7, Line 49: void setInputStream(BufferedInputStream in) {
Can this be in the constructor instead? Mutable state like this can get tricky. Same with the output stream below.
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@98
PS6, Line 98: interrupted = true;
> I'm asking whether that's what we're doing. 'finally' blocks are run uncond
Is there a test case around this retry/finally handling that verifies/show the expected behavior?
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/6/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@69
PS6, Line 69: String confFilePath = cmd.getOptionValue(CONF_LONG_OPTION);
> I'm imagining how this would work end-to-end and using config files, I thin
I think I agree with Andrew that we could start with command line arguments and grow into full on config files if needed.
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@35
PS7, Line 35: private static final int MAX_WRITER_THREAD_DEFAULT = 20;
This number feels really high to me, especially as a default. Is there any background to this choice?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 7
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 18 Dec 2019 19:49:41 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 13:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Executor.java@48
PS12, Line 48:
> Maybe name it SubprocessExecutor? Subprocess is not available as the protob
Sure
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@195
PS12, Line 195: MessageReader reade
> Yeah, I understand, but I chose not to do that as it would be weird to plum
Sure, I agree that it doesn't feel great to intermingle test-only code with non-test code. My overall feedback on this test is that it's pretty cluttered, and that makes it difficult to reason about what it's actually testing.
FWIW, if the goal is to test the specific behavior of MessageWriter, it might make sense to have tests that only instantiate a MessageWriter and a blocking queue (ie no Executor), and then manually inserting to the blocking queue and testing the behavior of the MessageWriter.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@210
PS12, Line 210: CompletableFuture writerFuture =
: CompletableFuture.runAsync(writer, writerService);
I don't really follow this comment. Mind rewriting it? Also probably don't need all the parens?
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestEchoSubprocess.java@214
PS12, Line 214:
:
> The writer task never timeout and will block on getting new messages.
Could we move the readerFuture.get() call out of the try/catch?
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@39
PS12, Line 39: public class TestMessageIO {
> So what is the purpose of such test in MessageIO? I thought we want that at
Having explicit tests for each component of a large system in isolation seems like a good idea. That way if the exception-handling of MessageIO ever changes for whatever reason, we don't have to debug more complex tests that involve entire Subprocesses -- we would see TestMessageIO fail and would clued into the fact that MessageIO is probably to blame.
http://gerrit.cloudera.org:8080/#/c/14329/12/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageIO.java@75
PS12, Line 75: size = MessageIO.intToBytes(100);
> Can you be more specific about what "isn't enough data" mean in this case?
Say we send over the following message:
[1000][10 bytes of stuff]
We should see an EOFException or something, right?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 13
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 03 Feb 2020 08:14:08 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14329
to look at the new patch set (#10).
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
[java] KUDU-2971: process communicates via protobuf-based protocol
This commit adds a java tool that can communicate over a stdin/stdout
pipe via protobuf-based protocol. It is useful in cases a Kudu process
(e.g. master) needs to talk to third-party libraries written in Java.
This tool has:
1) a single reader thread that continuously reads protobuf-based
messages from the standard input and puts the messages to a FIFO
blocking queue;
2) multiple writer threads that continuously retrieve the messages
from the queue, process them and write the responses to the standard
output.
IOException is fatal and causes the program to exit, e.g. I/O errors
when reading/writing to the pipe, and parsing malformed protobuf messages.
If encounter InterruptedException during placing/getting messages to/from
the queue, we consider it to be a signal to shutdown the task which
cause the program to exit as well.
To support a new protobuf message type, simply extend the 'ProtocolProcessor'
interface and add the specific ProcessorMain class (similar to 'EchoProcessorMain')
for the message type.
Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
---
A java/kudu-subprocess/build.gradle
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProcessorMain.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/KuduSubprocessException.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolProcessor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/Subprocessor.java
A java/kudu-subprocess/src/main/resources/log4j2.properties
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageProcessor.java
A java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java
M java/settings.gradle
16 files changed, 1,322 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/29/14329/10
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 10
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 5:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@89
PS5, Line 89: if (in.available() <= 0) {
can this be negative? anyway, we should return "" instead of new String() so that it can be cached.
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProcessor.java@125
PS5, Line 125: out.write(Bytes.concat(size, body));
: // Always do a flush after write to ensure no partial message is written.
: out.flush();
these two statements should be synchronized on out.
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@60
PS5, Line 60: true
this should be while (!Thread.currentThread().isInterrupted()). Right now there's no way to actually interrupt the thread as interrupted will be set to true, but we never get to finally.
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@94
PS5, Line 94: LOG.debug("Message: {} has been put on the queue", data);
this should be in an if (LOG.isDebugEnabled()) block so we don't format a string with potentially large data with needlessly when debug is turned off.
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@96
PS5, Line 96: } catch (InterruptedException e) {
why are we retrying when the thread is interrupted? Shouldn't we simply interrupt and immediately return?
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@63
PS5, Line 63: while (true) {
this should be while (!Thread.currentThread().isInterrupted()) as well, simply run Thread.currentThread().interrupt() in the catch block instead of setting the interrupted flag and then there's no need for the finally block.
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@69
PS5, Line 69: LOG.debug("Message: {} has been taken from the queue", data);
if (LOG.isDebugEnabled())
http://gerrit.cloudera.org:8080/#/c/14329/5/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java:
PS5:
we should register a shutdown hook that shuts down the threadpools (shutdownNow() to make sure the threads are interrupted)
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 5
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 14 Dec 2019 02:08:17 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has removed a vote on this change.
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 14:
(17 comments)
Overall looks much cleaner and simpler than the last revision I looked at. Thanks for making all of the changes.
Note: I didn't review the test code.
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolHandler.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolHandler.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolHandler.java@34
PS14, Line 34: resBuilder
Nit: elsewhere in the C++ code we generally use 'resp' as the prefix for the response rather than 'res'.
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@34
PS14, Line 34: * Util class for read and write protobuf message.
Nit: reading and writing protobuf messages.
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@98
PS14, Line 98: expect
expected
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java@105
PS14, Line 105: should
will
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@79
PS14, Line 79: exit
Nit: exiting
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@90
PS14, Line 90: if (data.length == 0) {
Under what circumstances would we expect to see an empty message? Doc?
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@37
PS14, Line 37: a
one
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@65
PS14, Line 65: while (!Thread.currentThread().isInterrupted()) {
Why do we have to test this condition? Why not just while (true)? If we're interrupted, won't it manifest as a thrown InterruptedException in the call to take()?
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@92
PS14, Line 92: SubprocessResponsePB.Builder res) {
Nit: resp
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java@123
PS14, Line 123: * Writes the response to the underlying output stream. IOException is fatal,
This method is simple enough that I think you can inline it into run().
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessCommandLineParser.java@91
PS14, Line 91: String queueSize = cmd.getOptionValue(QUEUE_SIZE_LONG_OPTION);
: String maxWriterThreads = cmd.getOptionValue(MAX_WRITER_THREADS_LONG_OPTION);
: String maxMsgBytes = cmd.getOptionValue(MAX_MESSAGE_BYTES_LONG_OPTION);
: conf = new SubprocessConfiguration(queueSize, maxWriterThreads, maxMsgBytes);
What do you think of passing the entire CommandLine into the SubprocessConfiguration constructor, and letting it pick out the specific option values it needs?
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@42
PS14, Line 42: Integer
Nit: add a space before
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java@44
PS14, Line 44:
Nit: maybe you can shorten the Javadoc for the below getters:
/**
* @return the blah blah blah, or the default value if not provided
*/
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@52
PS14, Line 52: @VisibleForTesting
This isn't actually "visible" for testing. The accessor is (and you've annotated it appropriately there), but the field itself is private as expected.
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@56
PS14, Line 56: // Exit the program with a nonzero status code if unexpected exception(s)
: // thrown by the reader or writer tasks.
: errorHandler = (t) -> {
: System.exit(1);
: return null;
: };
Can this be defined in the declaration of errorHandler itself? Could it be made static too?
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@108
PS14, Line 108: CompletableFuture[] writerFutures = new CompletableFuture[maxWriterThread];
Can you put the readerFuture into this array too? Then you can simplify the joining code below.
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java:
http://gerrit.cloudera.org:8080/#/c/14329/14/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessOutputStream.java@44
PS14, Line 44: @Override
Do we need to provide overrides for the other two write() variants, or for close()?
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 14
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 04 Feb 2020 22:02:22 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14329/7/java/kudu-subprocess/src/main/resources/log4j2.properties
File java/kudu-subprocess/src/main/resources/log4j2.properties:
PS7:
> I was thinking to use it as the default log config for the subprocess. Alth
Answering my own question: unlike our other modules (which basically provide libraries), this is a full-fledged application. Users will want to control the logging of the libraries, but we get to dictate how our application logs.
This was also the case for kudu-backup-tools, which has a main() and is an application in its own right.
Anyway, disregard this comment; makes sense to avoid debug-level o.a.k logging.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 8
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 15 Jan 2020 00:15:18 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2791: process communicates via protobuf-based protocol
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2791: process communicates via protobuf-based protocol
......................................................................
Patch Set 1:
(8 comments)
I did a quick first pass. I think the overall approach looks good.
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14329/1//COMMIT_MSG@15
PS1, Line 15: (with optional JSON encoding)
> Looked at your other patch, and I now get what you mean that you want to ke
I agree I don't see a strong need for it. We could remove all the configuration code and json code without it. It also simplifies the test matrix.
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/build.gradle
File java/kudu-subprocess/build.gradle:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/build.gradle@26
PS2, Line 26: compile libs.slf4jsimple
What is slf4jsimple required for?
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/EchoProtocolProcessor.java@27
PS2, Line 27: // Unless required by applicable law or agreed to in writing,
Should we added Private annotations to all of these classes to make sure no ones tries to use them?
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageConsumer.java@95
PS2, Line 95: break;
Maybe use a while loop? A for loop without an index initialization is fairly uncommon.
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageProducer.java@96
PS2, Line 96: try {
Is this an exceptional occurrence? When would the message be empty?
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMain.java@41
PS2, Line 41: private static final int QUEUE_SIZE = 100;
How was this size chosen?
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessRuntimeException.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessRuntimeException.java:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessRuntimeException.java@23
PS2, Line 23: final class SubprocessRuntimeException extends RuntimeException {
Maybe make this clearly Kudu related. `KuduSubprocessException`.
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/resources/log4j.properties
File java/kudu-subprocess/src/main/resources/log4j.properties:
http://gerrit.cloudera.org:8080/#/c/14329/2/java/kudu-subprocess/src/main/resources/log4j.properties@1
PS2, Line 1: log4j.debug=true
Match the log4j2.properties files like in all other modules.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 1
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 21 Oct 2019 16:03:06 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] KUDU-2971: process communicates via protobuf-based protocol
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14329 )
Change subject: [java] KUDU-2971: process communicates via protobuf-based protocol
......................................................................
Patch Set 12:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java
File java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java:
http://gerrit.cloudera.org:8080/#/c/14329/10/java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/TestMessageReaderWriter.java@119
PS10, Line 119:
:
:
:
:
:
:
:
:
:
:
:
> nit: IMO this makes this test significantly more difficult to read.
Yeah, makes sense. I refactored the test to be multiple test cases for easier reading.
--
To view, visit http://gerrit.cloudera.org:8080/14329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf9ad24dbc9acc681284b6433836271b5b4c7982
Gerrit-Change-Number: 14329
Gerrit-PatchSet: 12
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 27 Jan 2020 07:51:06 +0000
Gerrit-HasComments: Yes