You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2021/01/07 18:45:08 UTC
[GitHub] [nifi] pgyori opened a new pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
pgyori opened a new pull request #4748:
URL: https://github.com/apache/nifi/pull/4748
…he 'start' command
#### Description of PR
Enables starting NiFi with "bin/nifi.sh start --wait-for-init N" (where N is a positive integer) in which case the startup script will wait (up to N seconds) for NiFi to finish loading all components before returning.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-XXXX** where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r566884363
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -763,6 +773,27 @@ private void makeRequest(final String request, final String arguments, final Fil
}
}
+ private void sendRequest(Socket socket, Integer port, String request, String arguments, Logger logger) throws IOException {
+ logger.debug("Connecting to NiFi instance");
+ socket.setSoTimeout(60000);
+ socket.connect(new InetSocketAddress("localhost", port));
+ logger.debug("Established connection to NiFi instance.");
+ socket.setSoTimeout(60000);
+
+ logger.debug("Sending " + request + " Command to port {}", port);
Review comment:
Could you use built-in formatting instead of concatenation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] pgyori commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
pgyori commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r565947760
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
##########
@@ -345,6 +359,31 @@ run() {
if [ "$1" = "start" ]; then
( eval "cd ${NIFI_HOME} && ${run_nifi_cmd}" & )> /dev/null 1>&-
+
+ if [ "$2" = "--wait-for-init" ]; then
+
+ declare -i wait_timeout
+ if is_nonzero_integer "$3" ; then
+ wait_timeout="$3"
+ else
+ wait_timeout=$WAIT_FOR_INIT_DEFAULT_TIMEOUT
+ fi
+
+ declare -i starttime=$(date +%s)
+ declare -i endtime=$starttime+$wait_timeout
+ declare -i current_time
+
+ is_nifi_loaded="false"
+ while [ "$is_nifi_loaded" = "false" ]; do
+ current_time=$(date +%s)
+ if (( current_time >= endtime )); then
+ echo "Exited the script due to --wait-for-init timeout"
+ break;
+ fi
+ is_nifi_loaded=$( eval "cd ${NIFI_HOME} && ${run_bootstrap_cmd} is_loaded" )
Review comment:
That's a good idea. The $( eval "... is_loaded" ) call also takes a little time (besides the 2-second sleep), so I introduced a time-based feedback "mechanism" that will write on the standard output approximately every 10 seconds.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] pgyori commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
pgyori commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r565940606
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-runtime/src/main/java/org/apache/nifi/BootstrapListener.java
##########
@@ -215,6 +220,12 @@ public void run() {
writeDiagnostics(socket.getOutputStream(), verbose);
break;
+ case IS_LOADED:
+ logger.info("Received IS_LOADED request from Bootstrap");
Review comment:
That's right, thanks! Modified it in the next commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] pgyori commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
pgyori commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r568828167
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -763,6 +773,27 @@ private void makeRequest(final String request, final String arguments, final Fil
}
}
+ private void sendRequest(Socket socket, Integer port, String request, String arguments, Logger logger) throws IOException {
+ logger.debug("Connecting to NiFi instance");
+ socket.setSoTimeout(60000);
+ socket.connect(new InetSocketAddress("localhost", port));
+ logger.debug("Established connection to NiFi instance.");
+ socket.setSoTimeout(60000);
+
+ logger.debug("Sending " + request + " Command to port {}", port);
+ final OutputStream socketOut = socket.getOutputStream();
+
+ final Properties nifiProps = loadProperties(logger);
+ final String secretKey = nifiProps.getProperty("secret.key");
Review comment:
No. Each running NiFi instance has 1 secret key. We should not rely on the instance variable, as the bootstrap process can (and in this wait-for-init feature it IS) started over and over again 'independently' from the running NiFi instance, so the value of the secret key needs to be loaded from the property file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
markap14 commented on pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#issuecomment-772559407
Great work @pgyori, this all looks good to me! Did some verification of corner cases such as causing nifi to die during startup, etc. All appears to work as expected. +1 merged to main. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r563786001
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-runtime/src/main/java/org/apache/nifi/BootstrapListener.java
##########
@@ -215,6 +220,12 @@ public void run() {
writeDiagnostics(socket.getOutputStream(), verbose);
break;
+ case IS_LOADED:
+ logger.info("Received IS_LOADED request from Bootstrap");
Review comment:
We should probably make this a DEBUG log. Since this is happening every 2 seconds, we don't really want to see it constantly in the logs.
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
##########
@@ -345,6 +359,31 @@ run() {
if [ "$1" = "start" ]; then
( eval "cd ${NIFI_HOME} && ${run_nifi_cmd}" & )> /dev/null 1>&-
+
+ if [ "$2" = "--wait-for-init" ]; then
+
+ declare -i wait_timeout
+ if is_nonzero_integer "$3" ; then
+ wait_timeout="$3"
+ else
+ wait_timeout=$WAIT_FOR_INIT_DEFAULT_TIMEOUT
+ fi
+
+ declare -i starttime=$(date +%s)
+ declare -i endtime=$starttime+$wait_timeout
+ declare -i current_time
+
+ is_nifi_loaded="false"
+ while [ "$is_nifi_loaded" = "false" ]; do
+ current_time=$(date +%s)
+ if (( current_time >= endtime )); then
+ echo "Exited the script due to --wait-for-init timeout"
+ break;
+ fi
+ is_nifi_loaded=$( eval "cd ${NIFI_HOME} && ${run_bootstrap_cmd} is_loaded" )
Review comment:
If NiFi takes a while to load, the user experience can leave the use wondering if anything is happening, or if something is 'stuck', etc. Probably best to provide some feedback here. Perhaps just an `echo NiFi has not fully initialized yet....` type of thing. Perhaps not every 2 seconds though. Maybe every 5 iterations (10 seconds) or so would make sense?
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -711,6 +716,25 @@ public void dump(final File dumpFile) throws IOException {
makeRequest(DUMP_CMD, null, dumpFile, "thread dump");
}
+ private boolean isNiFiFullyLoaded() throws IOException {
+ final Logger logger = defaultLogger;
+ final Integer port = getCurrentPort(logger);
+ if (port == null) {
+ logger.info("Apache NiFi is not currently running");
Review comment:
If NiFi fails to start, we see this in the logs. However, the `--wait-for-init` results in this constantly running, logging this over and over. If we encounter a case where NiFi is not running, it should not continue waiting for it to initialize, since we know that it will never complete.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] pgyori commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
pgyori commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r568806370
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -711,6 +716,25 @@ public void dump(final File dumpFile) throws IOException {
makeRequest(DUMP_CMD, null, dumpFile, "thread dump");
}
+ private boolean isNiFiFullyLoaded() throws IOException {
+ final Logger logger = defaultLogger;
+ final Integer port = getCurrentPort(logger);
+ if (port == null) {
+ logger.info("Apache NiFi is not currently running");
Review comment:
In this particular case, I would prefer to leave it like that. There are many log messages in this class that would need to be extracted as constants, and the part of the class where the constants are, would be 'littered' with the crowd of log messages. In this case, I think extracting the messages would decrease readability.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] pgyori commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
pgyori commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r568829441
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -1467,4 +1498,11 @@ public boolean isProcessRunning() {
return Boolean.TRUE.equals(processRunning);
}
}
+
+ private static class NiFiNotRunningException extends Exception {
+ @Override
+ public Throwable fillInStackTrace() {
Review comment:
Yes, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r566887798
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -1467,4 +1498,11 @@ public boolean isProcessRunning() {
return Boolean.TRUE.equals(processRunning);
}
}
+
+ private static class NiFiNotRunningException extends Exception {
+ @Override
+ public Throwable fillInStackTrace() {
Review comment:
Would you make this synchronized to match parent class implementation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r566892972
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-runtime/src/main/java/org/apache/nifi/BootstrapListener.java
##########
@@ -46,6 +46,7 @@
private volatile Listener listener;
private volatile ServerSocket serverSocket;
Review comment:
I know this is not your modification but this one is interesting: objects shouldn't be marked volatile. "Marking a mutable object field volatile means the object reference is volatile but the object itself is not, and other threads may not see updates to the object state... For mutable objects, the volatile should be removed, and some other method should be used to ensure thread-safety, such as synchronization, or ThreadLocal storage."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] pgyori commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
pgyori commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r568838143
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-runtime/src/main/java/org/apache/nifi/BootstrapListener.java
##########
@@ -46,6 +46,7 @@
private volatile Listener listener;
private volatile ServerSocket serverSocket;
Review comment:
As far as I know, in the cases above, it actually is the reference that needs to be volatile since the value of the reference is changed from multiple threads and we need to make sure that no matter which thread reads the object behind this reference, it gets to the current object and not an outdated previous instance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r566892972
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-runtime/src/main/java/org/apache/nifi/BootstrapListener.java
##########
@@ -46,6 +46,7 @@
private volatile Listener listener;
private volatile ServerSocket serverSocket;
Review comment:
I know this is not your modification but this one is interesting, so just pointing out: objects shouldn't be marked volatile. "Marking a mutable object field volatile means the object reference is volatile but the object itself is not, and other threads may not see updates to the object state... For mutable objects, the volatile should be removed, and some other method should be used to ensure thread-safety, such as synchronization, or ThreadLocal storage."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 closed pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
markap14 closed pull request #4748:
URL: https://github.com/apache/nifi/pull/4748
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] pgyori commented on pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
pgyori commented on pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#issuecomment-772677768
Thank you, @markap14 !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r566892972
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-runtime/src/main/java/org/apache/nifi/BootstrapListener.java
##########
@@ -46,6 +46,7 @@
private volatile Listener listener;
private volatile ServerSocket serverSocket;
Review comment:
I know this is not your modification but this one is interesting, so just pointing out: objects shouldn't be marked volatile. "Marking a mutable object field volatile means the object reference is volatile but the object itself is not, and other threads may not see updates to the object state... For mutable objects, the volatile should be removed, and some other method should be used to ensure thread-safety, such as synchronization, or ThreadLocal storage."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] pgyori commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
pgyori commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r568806370
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -711,6 +716,25 @@ public void dump(final File dumpFile) throws IOException {
makeRequest(DUMP_CMD, null, dumpFile, "thread dump");
}
+ private boolean isNiFiFullyLoaded() throws IOException {
+ final Logger logger = defaultLogger;
+ final Integer port = getCurrentPort(logger);
+ if (port == null) {
+ logger.info("Apache NiFi is not currently running");
Review comment:
In this particular case, I would prefer to leave it like that. There are many log messages in this class that would need to be extracted as constants, and the part of the class where the constants are, would be 'littered' with the crowd of log messages. In this case, I think extracting the messages would decrease readability.
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -763,6 +773,27 @@ private void makeRequest(final String request, final String arguments, final Fil
}
}
+ private void sendRequest(Socket socket, Integer port, String request, String arguments, Logger logger) throws IOException {
+ logger.debug("Connecting to NiFi instance");
+ socket.setSoTimeout(60000);
+ socket.connect(new InetSocketAddress("localhost", port));
+ logger.debug("Established connection to NiFi instance.");
+ socket.setSoTimeout(60000);
+
+ logger.debug("Sending " + request + " Command to port {}", port);
Review comment:
That's right. The fix will be in my next commit.
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -763,6 +773,27 @@ private void makeRequest(final String request, final String arguments, final Fil
}
}
+ private void sendRequest(Socket socket, Integer port, String request, String arguments, Logger logger) throws IOException {
+ logger.debug("Connecting to NiFi instance");
+ socket.setSoTimeout(60000);
+ socket.connect(new InetSocketAddress("localhost", port));
+ logger.debug("Established connection to NiFi instance.");
+ socket.setSoTimeout(60000);
+
+ logger.debug("Sending " + request + " Command to port {}", port);
+ final OutputStream socketOut = socket.getOutputStream();
+
+ final Properties nifiProps = loadProperties(logger);
+ final String secretKey = nifiProps.getProperty("secret.key");
Review comment:
No. Each running NiFi instance has 1 secret key. We should not rely on the instance variable, as the bootstrap process can (and in this wait-for-init feature it IS) started over and over again 'independently' from the running NiFi instance, so the value of the secret key needs to be loaded from the property file.
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -1467,4 +1498,11 @@ public boolean isProcessRunning() {
return Boolean.TRUE.equals(processRunning);
}
}
+
+ private static class NiFiNotRunningException extends Exception {
+ @Override
+ public Throwable fillInStackTrace() {
Review comment:
Yes, thanks!
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-runtime/src/main/java/org/apache/nifi/BootstrapListener.java
##########
@@ -46,6 +46,7 @@
private volatile Listener listener;
private volatile ServerSocket serverSocket;
Review comment:
As far as I know, in the cases above, it actually is the reference that needs to be volatile since the value of the reference is changed from multiple threads and we need to make sure that no matter which thread reads the object behind this reference, it gets to the current object and not an outdated previous instance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] pgyori commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
pgyori commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r568817806
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -763,6 +773,27 @@ private void makeRequest(final String request, final String arguments, final Fil
}
}
+ private void sendRequest(Socket socket, Integer port, String request, String arguments, Logger logger) throws IOException {
+ logger.debug("Connecting to NiFi instance");
+ socket.setSoTimeout(60000);
+ socket.connect(new InetSocketAddress("localhost", port));
+ logger.debug("Established connection to NiFi instance.");
+ socket.setSoTimeout(60000);
+
+ logger.debug("Sending " + request + " Command to port {}", port);
Review comment:
That's right. The fix will be in my next commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r566885560
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -763,6 +773,27 @@ private void makeRequest(final String request, final String arguments, final Fil
}
}
+ private void sendRequest(Socket socket, Integer port, String request, String arguments, Logger logger) throws IOException {
+ logger.debug("Connecting to NiFi instance");
+ socket.setSoTimeout(60000);
+ socket.connect(new InetSocketAddress("localhost", port));
+ logger.debug("Established connection to NiFi instance.");
+ socket.setSoTimeout(60000);
+
+ logger.debug("Sending " + request + " Command to port {}", port);
+ final OutputStream socketOut = socket.getOutputStream();
+
+ final Properties nifiProps = loadProperties(logger);
+ final String secretKey = nifiProps.getProperty("secret.key");
Review comment:
secretKey hides the variable at row 124, can it cause any problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r563786001
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-runtime/src/main/java/org/apache/nifi/BootstrapListener.java
##########
@@ -215,6 +220,12 @@ public void run() {
writeDiagnostics(socket.getOutputStream(), verbose);
break;
+ case IS_LOADED:
+ logger.info("Received IS_LOADED request from Bootstrap");
Review comment:
We should probably make this a DEBUG log. Since this is happening every 2 seconds, we don't really want to see it constantly in the logs.
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
##########
@@ -345,6 +359,31 @@ run() {
if [ "$1" = "start" ]; then
( eval "cd ${NIFI_HOME} && ${run_nifi_cmd}" & )> /dev/null 1>&-
+
+ if [ "$2" = "--wait-for-init" ]; then
+
+ declare -i wait_timeout
+ if is_nonzero_integer "$3" ; then
+ wait_timeout="$3"
+ else
+ wait_timeout=$WAIT_FOR_INIT_DEFAULT_TIMEOUT
+ fi
+
+ declare -i starttime=$(date +%s)
+ declare -i endtime=$starttime+$wait_timeout
+ declare -i current_time
+
+ is_nifi_loaded="false"
+ while [ "$is_nifi_loaded" = "false" ]; do
+ current_time=$(date +%s)
+ if (( current_time >= endtime )); then
+ echo "Exited the script due to --wait-for-init timeout"
+ break;
+ fi
+ is_nifi_loaded=$( eval "cd ${NIFI_HOME} && ${run_bootstrap_cmd} is_loaded" )
Review comment:
If NiFi takes a while to load, the user experience can leave the use wondering if anything is happening, or if something is 'stuck', etc. Probably best to provide some feedback here. Perhaps just an `echo NiFi has not fully initialized yet....` type of thing. Perhaps not every 2 seconds though. Maybe every 5 iterations (10 seconds) or so would make sense?
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -711,6 +716,25 @@ public void dump(final File dumpFile) throws IOException {
makeRequest(DUMP_CMD, null, dumpFile, "thread dump");
}
+ private boolean isNiFiFullyLoaded() throws IOException {
+ final Logger logger = defaultLogger;
+ final Integer port = getCurrentPort(logger);
+ if (port == null) {
+ logger.info("Apache NiFi is not currently running");
Review comment:
If NiFi fails to start, we see this in the logs. However, the `--wait-for-init` results in this constantly running, logging this over and over. If we encounter a case where NiFi is not running, it should not continue waiting for it to initialize, since we know that it will never complete.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] pgyori commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
pgyori commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r566105108
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -711,6 +716,25 @@ public void dump(final File dumpFile) throws IOException {
makeRequest(DUMP_CMD, null, dumpFile, "thread dump");
}
+ private boolean isNiFiFullyLoaded() throws IOException {
+ final Logger logger = defaultLogger;
+ final Integer port = getCurrentPort(logger);
+ if (port == null) {
+ logger.info("Apache NiFi is not currently running");
Review comment:
I modified the code so that when the isNiFiFullyLoaded() method runs into this if branch, the nifi.sh script will detect that NiFi hasn't been started so there is no point in waiting. This logic is built on the premise that (port == null) means that NiFi hasn't been started. However, I found that when calling 'bin/nifi.sh start --wait-for-init', the first time isNiFiFullyLoaded() is executed happens before NiFi manages to reserve itself a port. So in my current solution (next commit) in the nifi.sh script we only assume that NiFi hasn't even been started if isNiFiFullyLoaded() runs into (port == null) 3 times (2 seconds pass between each call). Of course, it can result in exiting the nifi.sh script if it takes more than 4 seconds for NiFi to reserve a port.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a change in pull request #4748: NIFI-8123: Added support for --wait-for-init when NiFi started with t…
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on a change in pull request #4748:
URL: https://github.com/apache/nifi/pull/4748#discussion_r566882849
##########
File path: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java
##########
@@ -711,6 +716,25 @@ public void dump(final File dumpFile) throws IOException {
makeRequest(DUMP_CMD, null, dumpFile, "thread dump");
}
+ private boolean isNiFiFullyLoaded() throws IOException {
+ final Logger logger = defaultLogger;
+ final Integer port = getCurrentPort(logger);
+ if (port == null) {
+ logger.info("Apache NiFi is not currently running");
Review comment:
Would you consider to extract the log message into a constant as it is duplicated at 746. and 815. rows?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org