You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by C DINESH <di...@gmail.com> on 2020/05/23 13:55:15 UTC
Stateful-fun-Basic-Hello
Hi Team,
I am writing my first stateful fun basic hello application. I am getting
the following Exception.
$ ./bin/flink run -c
org.apache.flink.statefun.flink.core.StatefulFunctionsJob
./stateful-sun-hello-java-1.0-SNAPSHOT-jar-with-dependencies.jar
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: Invalid configuration:
classloader.parent-first-patterns.additional; Must contain all of
org.apache.flink.statefun, org.apache.kafka, com.google.protobuf
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:662)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:893)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:966)
at
org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:966)
Caused by:
org.apache.flink.statefun.flink.core.exceptions.StatefulFunctionsInvalidConfigException:
Invalid configuration: classloader.parent-first-patterns.additional; Must
contain all of org.apache.flink.statefun, org.apache.kafka,
com.google.protobuf
at
org.apache.flink.statefun.flink.core.StatefulFunctionsConfigValidator.validateParentFirstClassloaderPatterns(StatefulFunctionsConfigValidator.java:55)
at
org.apache.flink.statefun.flink.core.StatefulFunctionsConfigValidator.validate(StatefulFunctionsConfigValidator.java:44)
at
org.apache.flink.statefun.flink.core.StatefulFunctionsConfig.<init>(StatefulFunctionsConfig.java:143)
at
org.apache.flink.statefun.flink.core.StatefulFunctionsConfig.fromEnvironment(StatefulFunctionsConfig.java:105)
This is my POM file I hope I have added all the dependencies. Please
suggest me what to do.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>stateful-sun-hello-java</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.6.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>statefun-sdk</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>statefun-flink-distribution</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>statefun-kafka-io</artifactId>
<version>2.0.0</version>
</dependency>
</dependencies>
<build>
<defaultGoal>clean generate-sources compile install</defaultGoal>
<plugins>
<!-- compile proto file into java files. -->
<plugin>
<groupId>com.github.os72</groupId>
<artifactId>protoc-jar-maven-plugin</artifactId>
<version>3.6.0.1</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<includeMavenTypes>direct</includeMavenTypes>
<inputDirectories>
<include>src/main/protobuf</include>
</inputDirectories>
<outputTargets>
<outputTarget>
<type>java</type>
<outputDirectory>src/main/java</outputDirectory>
</outputTarget>
<outputTarget>
<type>grpc-java</type>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:1.15.0</pluginArtifact>
<outputDirectory>src/main/java</outputDirectory>
</outputTarget>
</outputTargets>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<!-- get all project dependencies -->
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<!-- MainClass in mainfest make a executable jar -->
<archive>
<manifest>
<mainClass>org.apache.flink.statefun.flink.core.StatefulFunctionsJob</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<!-- bind to the packaging phase -->
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
Thanks,
Dinesh
Re: Stateful-fun-Basic-Hello
Posted by C DINESH <di...@gmail.com>.
Thanks Gordan and Igal,
I understood the services file now. It is working now.
cheers,
Dinesh.
On Tue, May 26, 2020 at 5:50 PM C DINESH <di...@gmail.com> wrote:
> Hi Igal,
>
> org.apache.flink.statefun.sdk.spi.StatefulFunctionModule is not there in
> META-INF/services/
>
> [image: Screenshot 2020-05-26 at 5.46.35 PM.png]
>
> Here I am attaching the POM file. Please suggest me how can i fix.
>
>
>
> On Tue, May 26, 2020 at 3:44 PM Igal Shilman <ig...@ververica.com> wrote:
>
>> Hi,
>> Can you verify that your jar contains the following file
>> META-INF/services/org.apache.flink.statefun.sdk.spi.StatefulFunctionModule ?
>>
>> Thanks,
>> Igal.
>>
>> On Tue, May 26, 2020 at 11:49 AM C DINESH <di...@gmail.com>
>> wrote:
>>
>>> Hi Gordon,
>>>
>>> Thanks for your response.
>>>
>>> After adding this conf to flink-yml.
>>>
>>> `classloader.parent-first-patterns.additional:
>>> org.apache.flink.statefun;org.apache.kafka;com.google.protobuf`
>>>
>>> It gave me one more error
>>>
>>> The main method caused an error: Invalid configuration:
>>> jobmanager.scheduler; Currently the only supported scheduler is 'legacy'
>>>
>>> I updated to
>>>
>>> jobmanager.scheduler : legacy
>>>
>>> in flink-conf.yaml
>>>
>>> But know I got one more error. Which is self-explanatory. But actually I
>>> have provided ingress and egress in the module. I have attached a screen
>>> shot of my code. Please suggest me what to do.
>>>
>>> $ ./bin/flink run -c
>>> org.apache.flink.statefun.flink.core.StatefulFunctionsJob
>>> /Users/dineshchiramana/learning/flink_learning/stateful-fun-hello-java/target/stateful-fun-hello-java-1.0-SNAPSHOT-jar-with-dependencies.jar
>>>
>>>
>>>
>>> ------------------------------------------------------------
>>>
>>> The program finished with the following exception:
>>>
>>>
>>> org.apache.flink.client.program.ProgramInvocationException: The main
>>> method caused an error: There are no ingress defined.
>>>
>>> at
>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
>>>
>>> at
>>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
>>>
>>> at
>>> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
>>>
>>> at
>>> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:662)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>
>>> at
>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:893)
>>>
>>> at
>>> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:966)
>>>
>>> at
>>> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:966)
>>>
>>> Caused by: java.lang.IllegalStateException: There are no ingress defined.
>>>
>>> at
>>> org.apache.flink.statefun.flink.core.StatefulFunctionsUniverseValidator.validate(StatefulFunctionsUniverseValidator.java:25)
>>>
>>> at
>>> org.apache.flink.statefun.flink.core.StatefulFunctionsJob.main(StatefulFunctionsJob.java:66)
>>>
>>> at
>>> org.apache.flink.statefun.flink.core.StatefulFunctionsJob.main(StatefulFunctionsJob.java:41)
>>>
>>> at
>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>>
>>> at
>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>
>>> at
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>
>>> at
>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
>>>
>>> ... 8 more
>>>
>>>
>>>
>>>
>>> Cheers,
>>> Dinesh.
>>>
>>>
>>> On Tue, May 26, 2020 at 9:59 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> You're right, maybe the documentation needs a bit more directions
>>>> there, especially for people who are newer to Flink.
>>>>
>>>> 1. How to increase parallelism
>>>>
>>>> There are two ways to do this. Either set the `parallelism.default`
>>>> also in the flink-conf.yaml, or use the -p command line option when
>>>> starting the application via packaged Docker images.
>>>>
>>>> 2. How to enable checkpointing
>>>>
>>>> You would have to set execution.checkpointing.mode and
>>>> execution.checkpointing.interval configs, also in flink-conf.yaml.
>>>>
>>>> For example, the mode can be set to `EXACTLY_ONCE` and interval to
>>>> `5sec` to have exactly-once mode checkpoints at 5 second intervals.
>>>>
>>>>
>>>> In general, the Statefun specific configurations are listed here [1].
>>>> All other configurations available in Flink are also available in
>>>> Stateful Functions as well.
>>>>
>>>> Cheers,
>>>> Gordon
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/deployment-and-operations/configurations.html
>>>>
>>>> On Tue, May 26, 2020, 11:42 AM C DINESH <di...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Team,
>>>>>
>>>>> I mean to say that know I understood. but in the documentation page
>>>>> flink-conf.yaml is not mentioned
>>>>>
>>>>> On Mon, May 25, 2020 at 7:18 PM C DINESH <di...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Gordon,
>>>>>>
>>>>>> I read the documentation several times. But I didn't understand at
>>>>>> that time, flink-conf.yaml is not there.
>>>>>>
>>>>>> can you please suggest
>>>>>> 1. how to increase parallelism
>>>>>> 2. how to give checkpoints to the job
>>>>>>
>>>>>> As far as I know there is no documentation regarding this. or Are
>>>>>> these features are not there yet?
>>>>>>
>>>>>> Cheers,
>>>>>> Dinesh.
>>>>>>
>>>>>
Re: Stateful-fun-Basic-Hello
Posted by C DINESH <di...@gmail.com>.
Hi Igal,
org.apache.flink.statefun.sdk.spi.StatefulFunctionModule is not there in
META-INF/services/
[image: Screenshot 2020-05-26 at 5.46.35 PM.png]
Here I am attaching the POM file. Please suggest me how can i fix.
On Tue, May 26, 2020 at 3:44 PM Igal Shilman <ig...@ververica.com> wrote:
> Hi,
> Can you verify that your jar contains the following file
> META-INF/services/org.apache.flink.statefun.sdk.spi.StatefulFunctionModule ?
>
> Thanks,
> Igal.
>
> On Tue, May 26, 2020 at 11:49 AM C DINESH <di...@gmail.com>
> wrote:
>
>> Hi Gordon,
>>
>> Thanks for your response.
>>
>> After adding this conf to flink-yml.
>>
>> `classloader.parent-first-patterns.additional:
>> org.apache.flink.statefun;org.apache.kafka;com.google.protobuf`
>>
>> It gave me one more error
>>
>> The main method caused an error: Invalid configuration:
>> jobmanager.scheduler; Currently the only supported scheduler is 'legacy'
>>
>> I updated to
>>
>> jobmanager.scheduler : legacy
>>
>> in flink-conf.yaml
>>
>> But know I got one more error. Which is self-explanatory. But actually I
>> have provided ingress and egress in the module. I have attached a screen
>> shot of my code. Please suggest me what to do.
>>
>> $ ./bin/flink run -c
>> org.apache.flink.statefun.flink.core.StatefulFunctionsJob
>> /Users/dineshchiramana/learning/flink_learning/stateful-fun-hello-java/target/stateful-fun-hello-java-1.0-SNAPSHOT-jar-with-dependencies.jar
>>
>>
>>
>> ------------------------------------------------------------
>>
>> The program finished with the following exception:
>>
>>
>> org.apache.flink.client.program.ProgramInvocationException: The main
>> method caused an error: There are no ingress defined.
>>
>> at
>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
>>
>> at
>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
>>
>> at
>> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:662)
>>
>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:893)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:966)
>>
>> at
>> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>
>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:966)
>>
>> Caused by: java.lang.IllegalStateException: There are no ingress defined.
>>
>> at
>> org.apache.flink.statefun.flink.core.StatefulFunctionsUniverseValidator.validate(StatefulFunctionsUniverseValidator.java:25)
>>
>> at
>> org.apache.flink.statefun.flink.core.StatefulFunctionsJob.main(StatefulFunctionsJob.java:66)
>>
>> at
>> org.apache.flink.statefun.flink.core.StatefulFunctionsJob.main(StatefulFunctionsJob.java:41)
>>
>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>>
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>
>> at
>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
>>
>> ... 8 more
>>
>>
>>
>>
>> Cheers,
>> Dinesh.
>>
>>
>> On Tue, May 26, 2020 at 9:59 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
>> wrote:
>>
>>> Hi,
>>>
>>> You're right, maybe the documentation needs a bit more directions there,
>>> especially for people who are newer to Flink.
>>>
>>> 1. How to increase parallelism
>>>
>>> There are two ways to do this. Either set the `parallelism.default` also
>>> in the flink-conf.yaml, or use the -p command line option when starting the
>>> application via packaged Docker images.
>>>
>>> 2. How to enable checkpointing
>>>
>>> You would have to set execution.checkpointing.mode and
>>> execution.checkpointing.interval configs, also in flink-conf.yaml.
>>>
>>> For example, the mode can be set to `EXACTLY_ONCE` and interval to
>>> `5sec` to have exactly-once mode checkpoints at 5 second intervals.
>>>
>>>
>>> In general, the Statefun specific configurations are listed here [1].
>>> All other configurations available in Flink are also available in
>>> Stateful Functions as well.
>>>
>>> Cheers,
>>> Gordon
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/deployment-and-operations/configurations.html
>>>
>>> On Tue, May 26, 2020, 11:42 AM C DINESH <di...@gmail.com>
>>> wrote:
>>>
>>>> Hi Team,
>>>>
>>>> I mean to say that know I understood. but in the documentation page
>>>> flink-conf.yaml is not mentioned
>>>>
>>>> On Mon, May 25, 2020 at 7:18 PM C DINESH <di...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks Gordon,
>>>>>
>>>>> I read the documentation several times. But I didn't understand at
>>>>> that time, flink-conf.yaml is not there.
>>>>>
>>>>> can you please suggest
>>>>> 1. how to increase parallelism
>>>>> 2. how to give checkpoints to the job
>>>>>
>>>>> As far as I know there is no documentation regarding this. or Are
>>>>> these features are not there yet?
>>>>>
>>>>> Cheers,
>>>>> Dinesh.
>>>>>
>>>>
Re: Stateful-fun-Basic-Hello
Posted by Igal Shilman <ig...@ververica.com>.
Hi,
Can you verify that your jar contains the following file
META-INF/services/org.apache.flink.statefun.sdk.spi.StatefulFunctionModule ?
Thanks,
Igal.
On Tue, May 26, 2020 at 11:49 AM C DINESH <di...@gmail.com> wrote:
> Hi Gordon,
>
> Thanks for your response.
>
> After adding this conf to flink-yml.
>
> `classloader.parent-first-patterns.additional:
> org.apache.flink.statefun;org.apache.kafka;com.google.protobuf`
>
> It gave me one more error
>
> The main method caused an error: Invalid configuration:
> jobmanager.scheduler; Currently the only supported scheduler is 'legacy'
>
> I updated to
>
> jobmanager.scheduler : legacy
>
> in flink-conf.yaml
>
> But know I got one more error. Which is self-explanatory. But actually I
> have provided ingress and egress in the module. I have attached a screen
> shot of my code. Please suggest me what to do.
>
> $ ./bin/flink run -c
> org.apache.flink.statefun.flink.core.StatefulFunctionsJob
> /Users/dineshchiramana/learning/flink_learning/stateful-fun-hello-java/target/stateful-fun-hello-java-1.0-SNAPSHOT-jar-with-dependencies.jar
>
>
>
> ------------------------------------------------------------
>
> The program finished with the following exception:
>
>
> org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error: There are no ingress defined.
>
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
>
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
>
> at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
>
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:662)
>
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:893)
>
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:966)
>
> at
> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:966)
>
> Caused by: java.lang.IllegalStateException: There are no ingress defined.
>
> at
> org.apache.flink.statefun.flink.core.StatefulFunctionsUniverseValidator.validate(StatefulFunctionsUniverseValidator.java:25)
>
> at
> org.apache.flink.statefun.flink.core.StatefulFunctionsJob.main(StatefulFunctionsJob.java:66)
>
> at
> org.apache.flink.statefun.flink.core.StatefulFunctionsJob.main(StatefulFunctionsJob.java:41)
>
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
>
> ... 8 more
>
>
>
>
> Cheers,
> Dinesh.
>
>
> On Tue, May 26, 2020 at 9:59 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
>> Hi,
>>
>> You're right, maybe the documentation needs a bit more directions there,
>> especially for people who are newer to Flink.
>>
>> 1. How to increase parallelism
>>
>> There are two ways to do this. Either set the `parallelism.default` also
>> in the flink-conf.yaml, or use the -p command line option when starting the
>> application via packaged Docker images.
>>
>> 2. How to enable checkpointing
>>
>> You would have to set execution.checkpointing.mode and
>> execution.checkpointing.interval configs, also in flink-conf.yaml.
>>
>> For example, the mode can be set to `EXACTLY_ONCE` and interval to `5sec`
>> to have exactly-once mode checkpoints at 5 second intervals.
>>
>>
>> In general, the Statefun specific configurations are listed here [1].
>> All other configurations available in Flink are also available in
>> Stateful Functions as well.
>>
>> Cheers,
>> Gordon
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/deployment-and-operations/configurations.html
>>
>> On Tue, May 26, 2020, 11:42 AM C DINESH <di...@gmail.com> wrote:
>>
>>> Hi Team,
>>>
>>> I mean to say that know I understood. but in the documentation page
>>> flink-conf.yaml is not mentioned
>>>
>>> On Mon, May 25, 2020 at 7:18 PM C DINESH <di...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Gordon,
>>>>
>>>> I read the documentation several times. But I didn't understand at that
>>>> time, flink-conf.yaml is not there.
>>>>
>>>> can you please suggest
>>>> 1. how to increase parallelism
>>>> 2. how to give checkpoints to the job
>>>>
>>>> As far as I know there is no documentation regarding this. or Are these
>>>> features are not there yet?
>>>>
>>>> Cheers,
>>>> Dinesh.
>>>>
>>>
Re: Stateful-fun-Basic-Hello
Posted by C DINESH <di...@gmail.com>.
Hi Gordon,
Thanks for your response.
After adding this conf to flink-yml.
`classloader.parent-first-patterns.additional:
org.apache.flink.statefun;org.apache.kafka;com.google.protobuf`
It gave me one more error
The main method caused an error: Invalid configuration:
jobmanager.scheduler; Currently the only supported scheduler is 'legacy'
I updated to
jobmanager.scheduler : legacy
in flink-conf.yaml
But know I got one more error. Which is self-explanatory. But actually I
have provided ingress and egress in the module. I have attached a screen
shot of my code. Please suggest me what to do.
$ ./bin/flink run -c
org.apache.flink.statefun.flink.core.StatefulFunctionsJob
/Users/dineshchiramana/learning/flink_learning/stateful-fun-hello-java/target/stateful-fun-hello-java-1.0-SNAPSHOT-jar-with-dependencies.jar
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: There are no ingress defined.
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:662)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:893)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:966)
at
org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:966)
Caused by: java.lang.IllegalStateException: There are no ingress defined.
at
org.apache.flink.statefun.flink.core.StatefulFunctionsUniverseValidator.validate(StatefulFunctionsUniverseValidator.java:25)
at
org.apache.flink.statefun.flink.core.StatefulFunctionsJob.main(StatefulFunctionsJob.java:66)
at
org.apache.flink.statefun.flink.core.StatefulFunctionsJob.main(StatefulFunctionsJob.java:41)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
... 8 more
Cheers,
Dinesh.
On Tue, May 26, 2020 at 9:59 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Hi,
>
> You're right, maybe the documentation needs a bit more directions there,
> especially for people who are newer to Flink.
>
> 1. How to increase parallelism
>
> There are two ways to do this. Either set the `parallelism.default` also
> in the flink-conf.yaml, or use the -p command line option when starting the
> application via packaged Docker images.
>
> 2. How to enable checkpointing
>
> You would have to set execution.checkpointing.mode and
> execution.checkpointing.interval configs, also in flink-conf.yaml.
>
> For example, the mode can be set to `EXACTLY_ONCE` and interval to `5sec`
> to have exactly-once mode checkpoints at 5 second intervals.
>
>
> In general, the Statefun specific configurations are listed here [1].
> All other configurations available in Flink are also available in Stateful
> Functions as well.
>
> Cheers,
> Gordon
>
> [1]
> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/deployment-and-operations/configurations.html
>
> On Tue, May 26, 2020, 11:42 AM C DINESH <di...@gmail.com> wrote:
>
>> Hi Team,
>>
>> I mean to say that know I understood. but in the documentation page
>> flink-conf.yaml is not mentioned
>>
>> On Mon, May 25, 2020 at 7:18 PM C DINESH <di...@gmail.com>
>> wrote:
>>
>>> Thanks Gordon,
>>>
>>> I read the documentation several times. But I didn't understand at that
>>> time, flink-conf.yaml is not there.
>>>
>>> can you please suggest
>>> 1. how to increase parallelism
>>> 2. how to give checkpoints to the job
>>>
>>> As far as I know there is no documentation regarding this. or Are these
>>> features are not there yet?
>>>
>>> Cheers,
>>> Dinesh.
>>>
>>
Re: Stateful-fun-Basic-Hello
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi,
You're right, maybe the documentation needs a bit more directions there,
especially for people who are newer to Flink.
1. How to increase parallelism
There are two ways to do this. Either set the `parallelism.default` also in
the flink-conf.yaml, or use the -p command line option when starting the
application via packaged Docker images.
2. How to enable checkpointing
You would have to set execution.checkpointing.mode and
execution.checkpointing.interval configs, also in flink-conf.yaml.
For example, the mode can be set to `EXACTLY_ONCE` and interval to `5sec`
to have exactly-once mode checkpoints at 5 second intervals.
In general, the Statefun specific configurations are listed here [1].
All other configurations available in Flink are also available in Stateful
Functions as well.
Cheers,
Gordon
[1]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/deployment-and-operations/configurations.html
On Tue, May 26, 2020, 11:42 AM C DINESH <di...@gmail.com> wrote:
> Hi Team,
>
> I mean to say that know I understood. but in the documentation page
> flink-conf.yaml is not mentioned
>
> On Mon, May 25, 2020 at 7:18 PM C DINESH <di...@gmail.com> wrote:
>
>> Thanks Gordon,
>>
>> I read the documentation several times. But I didn't understand at that
>> time, flink-conf.yaml is not there.
>>
>> can you please suggest
>> 1. how to increase parallelism
>> 2. how to give checkpoints to the job
>>
>> As far as I know there is no documentation regarding this. or Are these
>> features are not there yet?
>>
>> Cheers,
>> Dinesh.
>>
>
Re: Stateful-fun-Basic-Hello
Posted by C DINESH <di...@gmail.com>.
Hi Team,
I mean to say that know I understood. but in the documentation page
flink-conf.yaml is not mentioned
On Mon, May 25, 2020 at 7:18 PM C DINESH <di...@gmail.com> wrote:
> Thanks Gordon,
>
> I read the documentation several times. But I didn't understand at that
> time, flink-conf.yaml is not there.
>
> can you please suggest
> 1. how to increase parallelism
> 2. how to give checkpoints to the job
>
> As far as I know there is no documentation regarding this. or Are these
> features are not there yet?
>
> Cheers,
> Dinesh.
>
Re: Stateful-fun-Basic-Hello
Posted by C DINESH <di...@gmail.com>.
Thanks Gordon,
I read the documentation several times. But I didn't understand at that
time, flink-conf.yaml is not there.
can you please suggest
1. how to increase parallelism
2. how to give checkpoints to the job
As far as I know there is no documentation regarding this. or Are these
features are not there yet?
Cheers,
Dinesh.
Re: Stateful-fun-Basic-Hello
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi,
It seems like you are trying to package your Stateful Functions app as a
Flink job, and submit that to an existing cluster.
If that indeed is the case,
Stateful Functions apps have some required confogurations that need to be
set via the flink-conf.yaml file for your existing cluster. Please see the
bottom half of [1] for more details.
Cheers,
Gordon
[1]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/deployment-and-operations/packaging.html#flink-jar
On Sat, May 23, 2020, 9:55 PM C DINESH <di...@gmail.com> wrote:
> Hi Team,
>
> I am writing my first stateful fun basic hello application. I am getting
> the following Exception.
>
> $ ./bin/flink run -c
> org.apache.flink.statefun.flink.core.StatefulFunctionsJob
> ./stateful-sun-hello-java-1.0-SNAPSHOT-jar-with-dependencies.jar
>
>
>
> ------------------------------------------------------------
>
> The program finished with the following exception:
>
>
> org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error: Invalid configuration:
> classloader.parent-first-patterns.additional; Must contain all of
> org.apache.flink.statefun, org.apache.kafka, com.google.protobuf
>
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
>
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
>
> at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
>
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:662)
>
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:893)
>
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:966)
>
> at
> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:966)
>
> Caused by:
> org.apache.flink.statefun.flink.core.exceptions.StatefulFunctionsInvalidConfigException:
> Invalid configuration: classloader.parent-first-patterns.additional; Must
> contain all of org.apache.flink.statefun, org.apache.kafka,
> com.google.protobuf
>
> at
> org.apache.flink.statefun.flink.core.StatefulFunctionsConfigValidator.validateParentFirstClassloaderPatterns(StatefulFunctionsConfigValidator.java:55)
>
> at
> org.apache.flink.statefun.flink.core.StatefulFunctionsConfigValidator.validate(StatefulFunctionsConfigValidator.java:44)
>
> at
> org.apache.flink.statefun.flink.core.StatefulFunctionsConfig.<init>(StatefulFunctionsConfig.java:143)
>
> at
> org.apache.flink.statefun.flink.core.StatefulFunctionsConfig.fromEnvironment(StatefulFunctionsConfig.java:105)
>
> This is my POM file I hope I have added all the dependencies. Please
> suggest me what to do.
>
> <?xml version="1.0" encoding="UTF-8"?>
> <project xmlns="http://maven.apache.org/POM/4.0.0"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
> <modelVersion>4.0.0</modelVersion>
>
> <groupId>org.example</groupId>
> <artifactId>stateful-sun-hello-java</artifactId>
> <version>1.0-SNAPSHOT</version>
>
> <dependencies>
> <dependency>
> <groupId>com.google.protobuf</groupId>
> <artifactId>protobuf-java</artifactId>
> <version>3.6.1</version>
> </dependency>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>statefun-sdk</artifactId>
> <version>2.0.0</version>
> </dependency>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>statefun-flink-distribution</artifactId>
> <version>2.0.0</version>
> </dependency>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>statefun-kafka-io</artifactId>
> <version>2.0.0</version>
> </dependency>
> </dependencies>
>
>
>
> <build>
> <defaultGoal>clean generate-sources compile install</defaultGoal>
>
> <plugins>
> <!-- compile proto file into java files. -->
> <plugin>
> <groupId>com.github.os72</groupId>
> <artifactId>protoc-jar-maven-plugin</artifactId>
> <version>3.6.0.1</version>
> <executions>
> <execution>
> <phase>generate-sources</phase>
> <goals>
> <goal>run</goal>
> </goals>
> <configuration>
> <includeMavenTypes>direct</includeMavenTypes>
>
> <inputDirectories>
> <include>src/main/protobuf</include>
> </inputDirectories>
>
> <outputTargets>
> <outputTarget>
> <type>java</type>
> <outputDirectory>src/main/java</outputDirectory>
> </outputTarget>
> <outputTarget>
> <type>grpc-java</type>
> <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.15.0</pluginArtifact>
> <outputDirectory>src/main/java</outputDirectory>
> </outputTarget>
> </outputTargets>
> </configuration>
> </execution>
> </executions>
> </plugin>
>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-assembly-plugin</artifactId>
> <version>2.4.1</version>
> <configuration>
> <!-- get all project dependencies -->
> <descriptorRefs>
> <descriptorRef>jar-with-dependencies</descriptorRef>
> </descriptorRefs>
> <!-- MainClass in mainfest make a executable jar -->
> <archive>
> <manifest>
> <mainClass>org.apache.flink.statefun.flink.core.StatefulFunctionsJob</mainClass>
> </manifest>
> </archive>
>
> </configuration>
> <executions>
> <execution>
> <id>make-assembly</id>
> <!-- bind to the packaging phase -->
> <phase>package</phase>
> <goals>
> <goal>single</goal>
> </goals>
> </execution>
> </executions>
> </plugin>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-compiler-plugin</artifactId>
> <version>3.8.0</version>
> <configuration>
> <source>1.8</source>
> <target>1.8</target>
> </configuration>
> </plugin>
> </plugins>
> </build>
>
> </project>
>
>
> Thanks,
> Dinesh
>
>
>