You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/03/10 07:52:50 UTC

[GitHub] [flink-statefun] FilKarnicki opened a new pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

FilKarnicki opened a new pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307


   ### What is the purpose of the change
   The goal of this PR is to add the ability to define when a statefun job is run from an uber jar. This then disables the validation of  `classloader.parent-first-patterns.additional` flink-config.yaml settings.
   
   It is then up to the creator of the uber jar to make sure that statefun, kafka and protobuf versions don't clash.
   
   ### Main changes are:
   - a new EMBEDDED config option was added (statefun.embedded)
   - if configuration does _not_ have this EMBEDDED option, then classloader validation happens
   - test util for catching throwables from a runnable so that it can be inspected
   
   ### Verifying this change
   Added a test for when the statefun.embedded option is and isn't present
   
   Dependencies (does it add or upgrade a dependency): no
   The public API, i.e., is any changed class annotated with @public(Evolving): N/A
   The serializers: no
   The runtime per-record code paths (performance sensitive): no
   Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
   The S3 file system connector: no


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] igalshilman commented on a change in pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
igalshilman commented on a change in pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#discussion_r824676050



##########
File path: statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/StatefulFunctionsJob.java
##########
@@ -36,18 +35,8 @@
   private static final AtomicInteger FEEDBACK_INVOCATION_ID_SEQ = new AtomicInteger();
 
   public static void main(String... args) throws Exception {
-    ParameterTool parameterTool = ParameterTool.fromArgs(args);
-    Map<String, String> globalConfigurations = parameterTool.toMap();
-
-    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

Review comment:
       I'm not sure that this is true, as far as I can tell, you should be able to set the parallelism even without create the env manually.
   
   I think that[ this is the place](https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/ClientUtils.java#L82) where the magic happens in Flink, where the correct env is prepared and set as thread local.
   
   The best thing would be to simply try it out.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] FilKarnicki commented on a change in pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
FilKarnicki commented on a change in pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#discussion_r824543145



##########
File path: statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/StatefulFunctionsJob.java
##########
@@ -36,18 +35,8 @@
   private static final AtomicInteger FEEDBACK_INVOCATION_ID_SEQ = new AtomicInteger();
 
   public static void main(String... args) throws Exception {
-    ParameterTool parameterTool = ParameterTool.fromArgs(args);
-    Map<String, String> globalConfigurations = parameterTool.toMap();
-
-    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

Review comment:
       I'm worried that I won't be able to set (for example) checkpointing intervals per job, unless I create the `StreamExecutionEnvironment` using job args. Am I correct?
   
   Knowing the subtleties of `getExecutionEnvironment`, do you think that using args as parameters for `getExecutionEnvironment` only when 'statefun.embedded' is 'true' is also too much of a change? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] FilKarnicki commented on pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
FilKarnicki commented on pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#issuecomment-1064082956


   Hi @igalshilman 
   I created some additional changes to StatefulFunctionsJob because the args passed to the main method weren't used to create the streaming env, and they weren't used to validate the settings (meaning that the override for embedded mode wasn't seen by the validator). 
   
   I'm wondering if the new behaviour could be breaking to existing users... That said, I think that using passed args to generate the env is the right thing to do. Especially for embedded users, who will otherwise not be able to (for example) set checkpointing interval durations per job.
   
   Please let me know what you think
   Fil


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] igalshilman commented on a change in pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
igalshilman commented on a change in pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#discussion_r824676050



##########
File path: statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/StatefulFunctionsJob.java
##########
@@ -36,18 +35,8 @@
   private static final AtomicInteger FEEDBACK_INVOCATION_ID_SEQ = new AtomicInteger();
 
   public static void main(String... args) throws Exception {
-    ParameterTool parameterTool = ParameterTool.fromArgs(args);
-    Map<String, String> globalConfigurations = parameterTool.toMap();
-
-    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

Review comment:
       I'm not sure that this is true, as far as I can tell, you should be able to set the parallelism even without creating the env manually.
   
   I think that[ this is the place](https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/ClientUtils.java#L82) where the magic happens in Flink, where the correct env is prepared and set as thread local.
   
   The best thing would be to simply try it out.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] FilKarnicki commented on pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
FilKarnicki commented on pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#issuecomment-1063896454


   All done I believe. Thank you. Fil


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] FilKarnicki commented on pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
FilKarnicki commented on pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#issuecomment-1066493012


   Indeed, the CLI settings do get used by flink, but REST ones don't seem to be (not for statefun jobs). I'll create a separate jira.
   
   For the record/future reference, checkpointing worked fine with: 
   ```
   flink run -m jobmanager:8081 -Dexecution.checkpointing.interval=1000 -Dstate.backend=rocksdb -Dstate.checkpoint-storage=filesystem -Dstate.checkpoints.dir=file:///tmp/ -Dstatefun.embedded=true /tmp/flinkjob-1.0-SNAPSHOT.jar
   ```
   
   ( from: https://github.com/FilKarnicki/statefun-flinkjob/blob/master/docker-compose/docker-compose.yml )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] igalshilman merged pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
igalshilman merged pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] FilKarnicki commented on pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
FilKarnicki commented on pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#issuecomment-1063970673


   Sorry, I need a little bit  more time to ensure that uber jar users can pass the statefun.embedded config to the validator


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] FilKarnicki commented on a change in pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
FilKarnicki commented on a change in pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#discussion_r823549473



##########
File path: statefun-flink/statefun-flink-core/src/test/java/org/apache/flink/statefun/flink/core/StatefulFunctionsConfigValidatorTest.java
##########
@@ -0,0 +1,39 @@
+package org.apache.flink.statefun.flink.core;
+
+import static org.apache.flink.statefun.flink.core.TestUtils.shouldFail;
+import static org.hamcrest.Matchers.startsWith;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.flink.api.java.utils.ParameterTool;
+import org.junit.Test;
+
+public class StatefulFunctionsConfigValidatorTest {
+
+  @Test
+  public void validatingWithoutEmbeddedModeShouldFail() {
+    Map<String, String> map = new HashMap<>();
+
+    Optional<Throwable> maybeThrowable =
+        shouldFail(
+            () ->
+                StatefulFunctionsConfigValidator.validate(
+                    ParameterTool.fromMap(map).getConfiguration()));
+
+    assertTrue(maybeThrowable.isPresent());
+    assertThat(
+        maybeThrowable.get().getMessage(),
+        startsWith("Invalid configuration: classloader.parent-first-patterns.additional"));

Review comment:
       makes sense. done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] igalshilman commented on pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
igalshilman commented on pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#issuecomment-1066179984


   Hi @FilKarnicki, no problem at all, and thanks for keep working on this.
   If I'm not mistaken, you can set any configuration with the `-D` flag when submitting from the cli, not sure about the REST api. 
   and you would probably want to set the following:
   https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#checkpointing
   and RocksDB.
   
   I think that this is a bit of a different issue that the original issue that this PR is trying to solve, what do you think about concluding this one, and following up?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] FilKarnicki edited a comment on pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
FilKarnicki edited a comment on pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#issuecomment-1066493012


   Hi @igalshilman. Indeed, the CLI settings do get used by flink, but REST ones don't seem to be (not for statefun jobs). I'll create a separate jira.
   
   For the record/future reference, checkpointing worked fine with: 
   ```
   flink run -m jobmanager:8081 -Dexecution.checkpointing.interval=1000 -Dstate.backend=rocksdb -Dstate.checkpoint-storage=filesystem -Dstate.checkpoints.dir=file:///tmp/ -Dstatefun.embedded=true /tmp/flinkjob-1.0-SNAPSHOT.jar
   ```
   
   ( from: https://github.com/FilKarnicki/statefun-flinkjob/blob/master/docker-compose/docker-compose.yml )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] FilKarnicki edited a comment on pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
FilKarnicki edited a comment on pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#issuecomment-1066493012


   Hi @igalshilman. Indeed, the CLI settings do get used by flink, but REST ones don't seem to be (not for statefun jobs). I'll create a separate jira.[edit: [FLINK-26628](https://issues.apache.org/jira/browse/FLINK-26628)]
   
   For the record/future reference, checkpointing worked fine with: 
   ```
   flink run -m jobmanager:8081 -Dexecution.checkpointing.interval=1000 -Dstate.backend=rocksdb -Dstate.checkpoint-storage=filesystem -Dstate.checkpoints.dir=file:///tmp/ -Dstatefun.embedded=true /tmp/flinkjob-1.0-SNAPSHOT.jar
   ```
   
   ( from: https://github.com/FilKarnicki/statefun-flinkjob/blob/master/docker-compose/docker-compose.yml )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] igalshilman commented on a change in pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
igalshilman commented on a change in pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#discussion_r823516954



##########
File path: statefun-flink/statefun-flink-core/src/test/java/org/apache/flink/statefun/flink/core/StatefulFunctionsConfigValidatorTest.java
##########
@@ -0,0 +1,39 @@
+package org.apache.flink.statefun.flink.core;

Review comment:
       missing licence header.

##########
File path: statefun-flink/statefun-flink-core/src/test/java/org/apache/flink/statefun/flink/core/StatefulFunctionsConfigValidatorTest.java
##########
@@ -0,0 +1,39 @@
+package org.apache.flink.statefun.flink.core;
+
+import static org.apache.flink.statefun.flink.core.TestUtils.shouldFail;
+import static org.hamcrest.Matchers.startsWith;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.flink.api.java.utils.ParameterTool;
+import org.junit.Test;
+
+public class StatefulFunctionsConfigValidatorTest {
+
+  @Test
+  public void validatingWithoutEmbeddedModeShouldFail() {
+    Map<String, String> map = new HashMap<>();
+
+    Optional<Throwable> maybeThrowable =
+        shouldFail(
+            () ->
+                StatefulFunctionsConfigValidator.validate(
+                    ParameterTool.fromMap(map).getConfiguration()));
+
+    assertTrue(maybeThrowable.isPresent());
+    assertThat(
+        maybeThrowable.get().getMessage(),
+        startsWith("Invalid configuration: classloader.parent-first-patterns.additional"));

Review comment:
       I understand the intention with this test, but from my personal experience tests that assert on an exception message text are fragile.
   I think that a simple unit test with an `@Test(expect = ...)` should be enough.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] FilKarnicki removed a comment on pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
FilKarnicki removed a comment on pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#issuecomment-1063896454


   All done I believe. Thank you. Fil


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] FilKarnicki commented on pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
FilKarnicki commented on pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#issuecomment-1065215184


   @igalshilman Thank you for helping me out. Hopefully this current version is a little bit closer to what you'd like to see. One thing I'm struggling to get my head around is how `StatefulFunctionsConfig#globalConfigurations` affects the env. I don't see any usages on the getter of `globalConfigurations` except for in `Modules` 
   
   I tried setting checkpointing interval settings both with `-execution.checkpointing.interval 10sec` and `-statefun.module.global-config.execution.checkpointing.interval 10sec` but wasn't successful in making flink start checkpointing.
   
   Again, many thanks for your help so far
   Fil


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] FilKarnicki commented on pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
FilKarnicki commented on pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#issuecomment-1067127571


   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] FilKarnicki removed a comment on pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
FilKarnicki removed a comment on pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#issuecomment-1065215184


   @igalshilman Thank you for helping me out. Hopefully this current version is a little bit closer to what you'd like to see. One thing I'm struggling to get my head around is how `StatefulFunctionsConfig#globalConfigurations` affects the env. I don't see any usages on the getter of `globalConfigurations` except for in `Modules` 
   
   I tried setting checkpointing interval settings both with `-execution.checkpointing.interval 10sec` and `-statefun.module.global-config.execution.checkpointing.interval 10sec` but wasn't successful in making flink start checkpointing.
   
   Again, many thanks for your help so far
   Fil


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] igalshilman commented on a change in pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
igalshilman commented on a change in pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#discussion_r824473506



##########
File path: statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/StatefulFunctionsJob.java
##########
@@ -36,18 +35,8 @@
   private static final AtomicInteger FEEDBACK_INVOCATION_ID_SEQ = new AtomicInteger();
 
   public static void main(String... args) throws Exception {
-    ParameterTool parameterTool = ParameterTool.fromArgs(args);
-    Map<String, String> globalConfigurations = parameterTool.toMap();
-
-    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

Review comment:
       While I understand what you are trying to do here, I'm not feeling comfortable with how big of a change that is, because there are some subtitles with the way configuration and the execution env are assembled and interacting.
   For example, the StreamExecutionEnvironment.getExecutionEnvironment() is set via thread local, and thus shared with a previously (stack wise) configurations set.
   
   I think however that you have a good point about the command line arguments.
   
   Looking at `StatefulFunctionsConfig` what do you think about the following plan:
   1. Add a getter and a setter for embedded like the other known properties in that class.
   2. When I look at `addAllGlobalConfigurations()` I see that they are simply added to the map, I think that the same logic needs to be applied for these configuration exactly as the constructor of this class.
   This will allow overriding exactly the same configuration parameter via the command line.
   (The precedence should be: first flink-conf.yaml, and then command line arguments.
   3. Move the validation after obtaining a valid StateFunConfig.
       
   ```
   StatefulFunctionsConfig stateFunConfig =
           StatefulFunctionsConfig.fromFlinkConfiguration(flinkConfig);
       stateFunConfig.addAllGlobalConfigurations(globalConfigurations);
       stateFunConfig.setProvider(new StatefulFunctionsUniverses.ClassPathUniverseProvider());
       StatefulFunctionsConfigValidator.validate(stateFunConfig, flinkConfig);  // use isEmbedded internally to toggle the relevant validations
   ```
   
   
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink-statefun] FilKarnicki commented on pull request #307: [FLINK-26537][statefun] Allow disabling StatefulFunctionsConfigValida…

Posted by GitBox <gi...@apache.org>.
FilKarnicki commented on pull request #307:
URL: https://github.com/apache/flink-statefun/pull/307#issuecomment-1065878580


   Hi @igalshilman, thank you for your continued support, I really appreciate it. As the code stands now in this pull request, when I add some logging and run this job via the web client wtih `execution.checkpointing.interval` set to `1000`
   ![image](https://user-images.githubusercontent.com/35812498/158017759-098ef5a4-65c1-4157-9962-e871cbb686a8.png)
   I see that globalConfigurations does indeed contain my interval setting
   ```
   723d5f18b87c_dockercompose_jobmanager_1 | 2022-03-12 12:12:30,428 INFO  org.apache.flink.client.ClientUtils                          [] - Starting program (detached: true)
   723d5f18b87c_dockercompose_jobmanager_1 | 2022-03-12 12:12:30,429 WARN  org.apache.flink.statefun.flink.core.StatefulFunctionsJob    [] - args: --execution.checkpointing.interval,1000
   723d5f18b87c_dockercompose_jobmanager_1 | 2022-03-12 12:12:30,430 WARN  org.apache.flink.statefun.flink.core.StatefulFunctionsJob    [] - argsParameterTool: {"execution.checkpointing.interval":"1000"}
   723d5f18b87c_dockercompose_jobmanager_1 | 2022-03-12 12:12:30,433 WARN  org.apache.flink.statefun.flink.core.StatefulFunctionsJob    [] - flinkConfigFromEnv: {"blob.server.port":"6124","taskmanager.memory.process.size":"1728m","jobmanager.execution.failover-strategy":"region","jobmanager.rpc.address":"jobmanager","jobmanager.memory.off-heap.size":"134217728b","jobmanager.memory.jvm-overhead.min":"201326592b","execution.target":"embedded","jobmanager.memory.process.size":"1600m","web.tmpdir":"/tmp/flink-web-7c6f8090-b2ab-47e7-9053-5cf7718afd6f","jobmanager.rpc.port":"6123","query.server.port":"6125","execution.savepoint.ignore-unclaimed-state":"false","execution.attached":"false","pipeline.jars":"file:/tmp/flink-web-7c6f8090-b2ab-47e7-9053-5cf7718afd6f/flink-web-upload/3d7f46f1-9395-4bb3-902f-aa1864fae86c_flinkjob-1.0-SNAPSHOT.jar","parallelism.default":"1","taskmanager.numberOfTaskSlots":"1","pipeline.classpaths":"","jobmanager.memory.jvm-metaspace.size":"268435456b","classloader.
 parent-first-patterns.additional":"org.apache.flink.statefun;org.apache.kafka;com.google.protobuf","jobmanager.memory.heap.size":"1073741824b","jobmanager.memory.jvm-overhead.max":"201326592b"}
   723d5f18b87c_dockercompose_jobmanager_1 | 2022-03-12 12:12:30,436 WARN  org.apache.flink.statefun.flink.core.StatefulFunctionsJob    [] - statefunConfigGlobalConfigurations: {"execution.checkpointing.interval":"1000"}
   ```
   Should I expect the checkpointing to be enabled? It doesn't look like it is
   ![image](https://user-images.githubusercontent.com/35812498/158017822-c1be5a68-6062-4c98-b156-bd52711aefc1.png)
   
   The way that I'm testing this is:
   1. `mvn package` this project https://github.com/FilKarnicki/statefun-flinkjob/blob/master/pom.xml
   2. `docker-compose up` this yaml https://github.com/FilKarnicki/statefun-flinkjob/blob/master/docker-compose/docker-compose.yml
   3. run this shell script https://github.com/FilKarnicki/statefun-flinkjob/blob/master/statefunCoreWithConfigLogging/patchCore.sh
   4. upload `flinkjob-1.0-SNAPSHOT.jar` to http://localhost:8081/#/submit
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org