You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Sandeep khanzode <sa...@shiftright.ai> on 2021/08/04 03:54:17 UTC
Bloom Filter - RocksDB - LinkageError Classloading
Hello,
I tried to add the bloom filter functionality as mentioned here:
https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html <https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html>
rocksDbBackend.setRocksDBOptions(new RocksDBOptionsFactory() {
public DBOptions createDBOptions(DBOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
return currentOptions.setMaxOpenFiles(1024);
}
public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
BloomFilter bloomFilter = new BloomFilter();
handlesToClose.add(bloomFilter);
return currentOptions
.setTableFormatConfig(
new BlockBasedTableConfig().setFilter(bloomFilter));
}
});
This is in the main class where we setup in the StreamExecutionEnvironment …
I get ClassLoading errors due to that ...
Caused by: java.lang.LinkageError: loader constraint violation: loader org.apache.flink.util.ChildFirstClassLoader @1169afe1 wants to load class org.rocksdb.ColumnFamilyOptions. A different class with the same name was previously loaded by 'app'. (org.rocksdb.ColumnFamilyOptions is in unnamed module of loader 'app')
What is documented is to change the order to parent-first in the flink-conf.yaml … but then I get different issues for the basic/core Spring Framework classes not being serializable …
Any help will be appreciated.
Thanks,
Sandip
Re: Bloom Filter - RocksDB - LinkageError Classloading
Posted by Sandeep khanzode <sa...@shiftright.ai>.
Hi Yun,
Yes, I will check how these classes are getting included.
I do see the below coming in from a Kafka-Streams dependency being used for another module.
org.rocksdb:rocksdbjni:jar:5.18.4:compile
Thanks,
Sandip
> On 10-Aug-2021, at 8:22 AM, Yun Tang <my...@live.com> wrote:
>
> Hi Sandeep,
>
> I'm afraid that it's because the configuration of your maven plugins like maven-assembly or maven-shaded to include the classes.
>
> Best
> Yun Tang
> From: Sandeep khanzode <sa...@shiftright.ai>
> Sent: Saturday, August 7, 2021 10:20
> To: Yun Tang <my...@live.com>
> Cc: Stephan Ewen <se...@apache.org>; user <us...@flink.apache.org>
> Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading
>
> Hi Yun,
>
> You are right. This is what I see in my application JAR …
>
> Does this mean that the Rocks DB dependency is still getting packaged in my application JAR and this is what is causing the issue?
>
> I have made the scope as Provided for that dependency. Is it being transitively pulled in? All my Flink dependencies are set to Provided in Maven.
>
> <Screenshot 2021-08-07 at 7.47.58 AM.png>
>
> <Screenshot 2021-08-07 at 7.48.42 AM.png>
>
>> On 06-Aug-2021, at 7:58 AM, Yun Tang <myasuka@live.com <ma...@live.com>> wrote:
>>
>> Hi Sandeep,
>>
>> If you set the flink-statebackend-rocksdb as provided scope, it should not include the org.rocksdb classes, have you ever checked your application jar package directly just as what I described?
>>
>>
>> Best
>> Yun Tang
>> From: Sandeep khanzode <sandeep@shiftright.ai <ma...@shiftright.ai>>
>> Sent: Friday, August 6, 2021 2:04
>> To: Stephan Ewen <sewen@apache.org <ma...@apache.org>>
>> Cc: user <user@flink.apache.org <ma...@flink.apache.org>>; Yun Tang <myasuka@live.com <ma...@live.com>>
>> Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading
>>
>> Hello Stephan, Yun,
>>
>> Thanks for your insights.
>>
>> All I have added is this:
>> <dependency>
>> <groupId>org.apache.flink</groupId>
>> <artifactId>flink-statebackend-rocksdb_2.12</artifactId>
>> <version>${flink.version}</version>
>> <scope>provided</scope>
>> </dependency>
>>
>> No other library explicitly added. I am assuming, as mentioned, is that the flink-dist.jar already contains the relevant classes and the App or parent class loader loads the Rocks DB classes. All other Flink dependencies are packaged as Maven - Provided.
>>
>> Moving to parent-first gives the Spring Framework serialisation issues … I will take a look at that …
>>
>> I thought it would be simpler to simply specify Bloom Filters as an option …
>>
>> Maybe, I will have to remove Spring dependency …
>>
>>
>> Thanks,
>> Sandip
>>
>>
>>
>>> On 05-Aug-2021, at 5:55 PM, Stephan Ewen <sewen@apache.org <ma...@apache.org>> wrote:
>>>
>>> @Yun Tang
>>>
>>> Our FRocksDB has the same java package names (org.rocksdb.). Adding 'org.rocksdb' to parent-first patterns ensures it will be loaded only once, and not accidentally multiple times (as Child-first classloading does).
>>>
>>> The RocksDB code here is a bit like Flink internal components, which we always force parent-first (everything that starts with "org.apache.fink.").
>>>
>>> To use RocksDB from the application jar, I think you would need to remove the RocksDB state backend from the classpath (lib folder), or you get exactly the error reported above.
>>>
>>> I cannot think of a downside to add RocksDB to the parent first patterns.
>>>
>>> On Thu, Aug 5, 2021 at 10:04 AM Yun Tang <myasuka@live.com <ma...@live.com>> wrote:
>>> Hi Stephan,
>>>
>>> Since we use our own FRocksDB instead of the original RocksDB as dependency, I am not sure whether this problem has relationship with this. From my knowledge, more customers would include Flink classes within the application jar package, and it might cause problems if the client has different flink version with servers.
>>>
>>>
>>> Best,
>>> Yun Tang
>>> From: Stephan Ewen <sewen@apache.org <ma...@apache.org>>
>>> Sent: Wednesday, August 4, 2021 19:10
>>> To: Yun Tang <myasuka@live.com <ma...@live.com>>
>>> Cc: Sandeep khanzode <sandeep@shiftright.ai <ma...@shiftright.ai>>; user <user@flink.apache.org <ma...@flink.apache.org>>
>>> Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading
>>>
>>> @Yun Tang Does it make sense to add RocksDB to the "always parent-first options" to avoid these kind of errors when users package apps incorrectly?
>>> My feeling is that these packaging errors occur very frequently.
>>>
>>>
>>> On Wed, Aug 4, 2021 at 10:41 AM Yun Tang <myasuka@live.com <ma...@live.com>> wrote:
>>> Hi Sandeep,
>>>
>>> Did you include the RocksDB classes in the application jar package? You can unpark your jar package to check whether them existed.
>>> If so, since RocksDB classes are already included in the flink-dist package, you don't need to include them in your jar package (maybe you explicitly added the dependency of org.rocksdb:rocksdbjni in your pom).
>>>
>>> Best
>>> Yun Tang
>>> From: Sandeep khanzode <sandeep@shiftright.ai <ma...@shiftright.ai>>
>>> Sent: Wednesday, August 4, 2021 11:54
>>> To: user <user@flink.apache.org <ma...@flink.apache.org>>
>>> Subject: Bloom Filter - RocksDB - LinkageError Classloading
>>>
>>> Hello,
>>>
>>> I tried to add the bloom filter functionality as mentioned here:
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html <https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html>
>>>
>>> rocksDbBackend.setRocksDBOptions(new RocksDBOptionsFactory() {
>>>
>>> public DBOptions createDBOptions(DBOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
>>> return currentOptions.setMaxOpenFiles(1024);
>>> }
>>>
>>> public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
>>> BloomFilter bloomFilter = new BloomFilter();
>>> handlesToClose.add(bloomFilter);
>>>
>>> return currentOptions
>>> .setTableFormatConfig(
>>> new BlockBasedTableConfig().setFilter(bloomFilter));
>>> }
>>> });
>>>
>>> This is in the main class where we setup in the StreamExecutionEnvironment …
>>>
>>> I get ClassLoading errors due to that ...
>>> Caused by: java.lang.LinkageError: loader constraint violation: loader org.apache.flink.util.ChildFirstClassLoader @1169afe1 wants to load class org.rocksdb.ColumnFamilyOptions. A different class with the same name was previously loaded by 'app'. (org.rocksdb.ColumnFamilyOptions is in unnamed module of loader 'app')
>>>
>>>
>>> What is documented is to change the order to parent-first in the flink-conf.yaml … but then I get different issues for the basic/core Spring Framework classes not being serializable …
>>>
>>> Any help will be appreciated.
>>>
>>> Thanks,
>>> Sandip
Re: Bloom Filter - RocksDB - LinkageError Classloading
Posted by Yun Tang <my...@live.com>.
Hi Sandeep,
I'm afraid that it's because the configuration of your maven plugins like maven-assembly or maven-shaded to include the classes.
Best
Yun Tang
________________________________
From: Sandeep khanzode <sa...@shiftright.ai>
Sent: Saturday, August 7, 2021 10:20
To: Yun Tang <my...@live.com>
Cc: Stephan Ewen <se...@apache.org>; user <us...@flink.apache.org>
Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading
Hi Yun,
You are right. This is what I see in my application JAR …
Does this mean that the Rocks DB dependency is still getting packaged in my application JAR and this is what is causing the issue?
I have made the scope as Provided for that dependency. Is it being transitively pulled in? All my Flink dependencies are set to Provided in Maven.
[cid:CD686585-4C06-4923-8710-47098490A8EB]
[cid:26422E55-55BE-4642-878F-A89894C11311]
On 06-Aug-2021, at 7:58 AM, Yun Tang <my...@live.com>> wrote:
Hi Sandeep,
If you set the flink-statebackend-rocksdb as provided scope, it should not include the org.rocksdb classes, have you ever checked your application jar package directly just as what I described?
Best
Yun Tang
________________________________
From: Sandeep khanzode <sa...@shiftright.ai>>
Sent: Friday, August 6, 2021 2:04
To: Stephan Ewen <se...@apache.org>>
Cc: user <us...@flink.apache.org>>; Yun Tang <my...@live.com>>
Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading
Hello Stephan, Yun,
Thanks for your insights.
All I have added is this:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-statebackend-rocksdb_2.12</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
No other library explicitly added. I am assuming, as mentioned, is that the flink-dist.jar already contains the relevant classes and the App or parent class loader loads the Rocks DB classes. All other Flink dependencies are packaged as Maven - Provided.
Moving to parent-first gives the Spring Framework serialisation issues … I will take a look at that …
I thought it would be simpler to simply specify Bloom Filters as an option …
Maybe, I will have to remove Spring dependency …
Thanks,
Sandip
On 05-Aug-2021, at 5:55 PM, Stephan Ewen <se...@apache.org>> wrote:
@Yun Tang
Our FRocksDB has the same java package names (org.rocksdb.). Adding 'org.rocksdb' to parent-first patterns ensures it will be loaded only once, and not accidentally multiple times (as Child-first classloading does).
The RocksDB code here is a bit like Flink internal components, which we always force parent-first (everything that starts with "org.apache.fink.").
To use RocksDB from the application jar, I think you would need to remove the RocksDB state backend from the classpath (lib folder), or you get exactly the error reported above.
I cannot think of a downside to add RocksDB to the parent first patterns.
On Thu, Aug 5, 2021 at 10:04 AM Yun Tang <my...@live.com>> wrote:
Hi Stephan,
Since we use our own FRocksDB instead of the original RocksDB as dependency, I am not sure whether this problem has relationship with this. From my knowledge, more customers would include Flink classes within the application jar package, and it might cause problems if the client has different flink version with servers.
Best,
Yun Tang
________________________________
From: Stephan Ewen <se...@apache.org>>
Sent: Wednesday, August 4, 2021 19:10
To: Yun Tang <my...@live.com>>
Cc: Sandeep khanzode <sa...@shiftright.ai>>; user <us...@flink.apache.org>>
Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading
@Yun Tang Does it make sense to add RocksDB to the "always parent-first options" to avoid these kind of errors when users package apps incorrectly?
My feeling is that these packaging errors occur very frequently.
On Wed, Aug 4, 2021 at 10:41 AM Yun Tang <my...@live.com>> wrote:
Hi Sandeep,
Did you include the RocksDB classes in the application jar package? You can unpark your jar package to check whether them existed.
If so, since RocksDB classes are already included in the flink-dist package, you don't need to include them in your jar package (maybe you explicitly added the dependency of org.rocksdb:rocksdbjni in your pom).
Best
Yun Tang
________________________________
From: Sandeep khanzode <sa...@shiftright.ai>>
Sent: Wednesday, August 4, 2021 11:54
To: user <us...@flink.apache.org>>
Subject: Bloom Filter - RocksDB - LinkageError Classloading
Hello,
I tried to add the bloom filter functionality as mentioned here:
https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html
rocksDbBackend.setRocksDBOptions(new RocksDBOptionsFactory() {
public DBOptions createDBOptions(DBOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
return currentOptions.setMaxOpenFiles(1024);
}
public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
BloomFilter bloomFilter = new BloomFilter();
handlesToClose.add(bloomFilter);
return currentOptions
.setTableFormatConfig(
new BlockBasedTableConfig().setFilter(bloomFilter));
}
});
This is in the main class where we setup in the StreamExecutionEnvironment …
I get ClassLoading errors due to that ...
Caused by: java.lang.LinkageError: loader constraint violation: loader org.apache.flink.util.ChildFirstClassLoader @1169afe1 wants to load class org.rocksdb.ColumnFamilyOptions. A different class with the same name was previously loaded by 'app'. (org.rocksdb.ColumnFamilyOptions is in unnamed module of loader 'app')
What is documented is to change the order to parent-first in the flink-conf.yaml … but then I get different issues for the basic/core Spring Framework classes not being serializable …
Any help will be appreciated.
Thanks,
Sandip
Re: Bloom Filter - RocksDB - LinkageError Classloading
Posted by Sandeep khanzode <sa...@shiftright.ai>.
Hi Yun,
You are right. This is what I see in my application JAR …
Does this mean that the Rocks DB dependency is still getting packaged in my application JAR and this is what is causing the issue?
I have made the scope as Provided for that dependency. Is it being transitively pulled in? All my Flink dependencies are set to Provided in Maven.
> On 06-Aug-2021, at 7:58 AM, Yun Tang <my...@live.com> wrote:
>
> Hi Sandeep,
>
> If you set the flink-statebackend-rocksdb as provided scope, it should not include the org.rocksdb classes, have you ever checked your application jar package directly just as what I described?
>
>
> Best
> Yun Tang
> From: Sandeep khanzode <sa...@shiftright.ai>
> Sent: Friday, August 6, 2021 2:04
> To: Stephan Ewen <se...@apache.org>
> Cc: user <us...@flink.apache.org>; Yun Tang <my...@live.com>
> Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading
>
> Hello Stephan, Yun,
>
> Thanks for your insights.
>
> All I have added is this:
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-statebackend-rocksdb_2.12</artifactId>
> <version>${flink.version}</version>
> <scope>provided</scope>
> </dependency>
>
> No other library explicitly added. I am assuming, as mentioned, is that the flink-dist.jar already contains the relevant classes and the App or parent class loader loads the Rocks DB classes. All other Flink dependencies are packaged as Maven - Provided.
>
> Moving to parent-first gives the Spring Framework serialisation issues … I will take a look at that …
>
> I thought it would be simpler to simply specify Bloom Filters as an option …
>
> Maybe, I will have to remove Spring dependency …
>
>
> Thanks,
> Sandip
>
>
>
>> On 05-Aug-2021, at 5:55 PM, Stephan Ewen <sewen@apache.org <ma...@apache.org>> wrote:
>>
>> @Yun Tang
>>
>> Our FRocksDB has the same java package names (org.rocksdb.). Adding 'org.rocksdb' to parent-first patterns ensures it will be loaded only once, and not accidentally multiple times (as Child-first classloading does).
>>
>> The RocksDB code here is a bit like Flink internal components, which we always force parent-first (everything that starts with "org.apache.fink.").
>>
>> To use RocksDB from the application jar, I think you would need to remove the RocksDB state backend from the classpath (lib folder), or you get exactly the error reported above.
>>
>> I cannot think of a downside to add RocksDB to the parent first patterns.
>>
>> On Thu, Aug 5, 2021 at 10:04 AM Yun Tang <myasuka@live.com <ma...@live.com>> wrote:
>> Hi Stephan,
>>
>> Since we use our own FRocksDB instead of the original RocksDB as dependency, I am not sure whether this problem has relationship with this. From my knowledge, more customers would include Flink classes within the application jar package, and it might cause problems if the client has different flink version with servers.
>>
>>
>> Best,
>> Yun Tang
>> From: Stephan Ewen <sewen@apache.org <ma...@apache.org>>
>> Sent: Wednesday, August 4, 2021 19:10
>> To: Yun Tang <myasuka@live.com <ma...@live.com>>
>> Cc: Sandeep khanzode <sandeep@shiftright.ai <ma...@shiftright.ai>>; user <user@flink.apache.org <ma...@flink.apache.org>>
>> Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading
>>
>> @Yun Tang Does it make sense to add RocksDB to the "always parent-first options" to avoid these kind of errors when users package apps incorrectly?
>> My feeling is that these packaging errors occur very frequently.
>>
>>
>> On Wed, Aug 4, 2021 at 10:41 AM Yun Tang <myasuka@live.com <ma...@live.com>> wrote:
>> Hi Sandeep,
>>
>> Did you include the RocksDB classes in the application jar package? You can unpark your jar package to check whether them existed.
>> If so, since RocksDB classes are already included in the flink-dist package, you don't need to include them in your jar package (maybe you explicitly added the dependency of org.rocksdb:rocksdbjni in your pom).
>>
>> Best
>> Yun Tang
>> From: Sandeep khanzode <sandeep@shiftright.ai <ma...@shiftright.ai>>
>> Sent: Wednesday, August 4, 2021 11:54
>> To: user <user@flink.apache.org <ma...@flink.apache.org>>
>> Subject: Bloom Filter - RocksDB - LinkageError Classloading
>>
>> Hello,
>>
>> I tried to add the bloom filter functionality as mentioned here:
>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html <https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html>
>>
>> rocksDbBackend.setRocksDBOptions(new RocksDBOptionsFactory() {
>>
>> public DBOptions createDBOptions(DBOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
>> return currentOptions.setMaxOpenFiles(1024);
>> }
>>
>> public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
>> BloomFilter bloomFilter = new BloomFilter();
>> handlesToClose.add(bloomFilter);
>>
>> return currentOptions
>> .setTableFormatConfig(
>> new BlockBasedTableConfig().setFilter(bloomFilter));
>> }
>> });
>>
>> This is in the main class where we setup in the StreamExecutionEnvironment …
>>
>> I get ClassLoading errors due to that ...
>> Caused by: java.lang.LinkageError: loader constraint violation: loader org.apache.flink.util.ChildFirstClassLoader @1169afe1 wants to load class org.rocksdb.ColumnFamilyOptions. A different class with the same name was previously loaded by 'app'. (org.rocksdb.ColumnFamilyOptions is in unnamed module of loader 'app')
>>
>>
>> What is documented is to change the order to parent-first in the flink-conf.yaml … but then I get different issues for the basic/core Spring Framework classes not being serializable …
>>
>> Any help will be appreciated.
>>
>> Thanks,
>> Sandip
Re: Bloom Filter - RocksDB - LinkageError Classloading
Posted by Yun Tang <my...@live.com>.
Hi Sandeep,
If you set the flink-statebackend-rocksdb as provided scope, it should not include the org.rocksdb classes, have you ever checked your application jar package directly just as what I described?
Best
Yun Tang
________________________________
From: Sandeep khanzode <sa...@shiftright.ai>
Sent: Friday, August 6, 2021 2:04
To: Stephan Ewen <se...@apache.org>
Cc: user <us...@flink.apache.org>; Yun Tang <my...@live.com>
Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading
Hello Stephan, Yun,
Thanks for your insights.
All I have added is this:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-statebackend-rocksdb_2.12</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
No other library explicitly added. I am assuming, as mentioned, is that the flink-dist.jar already contains the relevant classes and the App or parent class loader loads the Rocks DB classes. All other Flink dependencies are packaged as Maven - Provided.
Moving to parent-first gives the Spring Framework serialisation issues … I will take a look at that …
I thought it would be simpler to simply specify Bloom Filters as an option …
Maybe, I will have to remove Spring dependency …
Thanks,
Sandip
On 05-Aug-2021, at 5:55 PM, Stephan Ewen <se...@apache.org>> wrote:
@Yun Tang
Our FRocksDB has the same java package names (org.rocksdb.). Adding 'org.rocksdb' to parent-first patterns ensures it will be loaded only once, and not accidentally multiple times (as Child-first classloading does).
The RocksDB code here is a bit like Flink internal components, which we always force parent-first (everything that starts with "org.apache.fink.").
To use RocksDB from the application jar, I think you would need to remove the RocksDB state backend from the classpath (lib folder), or you get exactly the error reported above.
I cannot think of a downside to add RocksDB to the parent first patterns.
On Thu, Aug 5, 2021 at 10:04 AM Yun Tang <my...@live.com>> wrote:
Hi Stephan,
Since we use our own FRocksDB instead of the original RocksDB as dependency, I am not sure whether this problem has relationship with this. From my knowledge, more customers would include Flink classes within the application jar package, and it might cause problems if the client has different flink version with servers.
Best,
Yun Tang
________________________________
From: Stephan Ewen <se...@apache.org>>
Sent: Wednesday, August 4, 2021 19:10
To: Yun Tang <my...@live.com>>
Cc: Sandeep khanzode <sa...@shiftright.ai>>; user <us...@flink.apache.org>>
Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading
@Yun Tang Does it make sense to add RocksDB to the "always parent-first options" to avoid these kind of errors when users package apps incorrectly?
My feeling is that these packaging errors occur very frequently.
On Wed, Aug 4, 2021 at 10:41 AM Yun Tang <my...@live.com>> wrote:
Hi Sandeep,
Did you include the RocksDB classes in the application jar package? You can unpark your jar package to check whether them existed.
If so, since RocksDB classes are already included in the flink-dist package, you don't need to include them in your jar package (maybe you explicitly added the dependency of org.rocksdb:rocksdbjni in your pom).
Best
Yun Tang
________________________________
From: Sandeep khanzode <sa...@shiftright.ai>>
Sent: Wednesday, August 4, 2021 11:54
To: user <us...@flink.apache.org>>
Subject: Bloom Filter - RocksDB - LinkageError Classloading
Hello,
I tried to add the bloom filter functionality as mentioned here:
https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html
rocksDbBackend.setRocksDBOptions(new RocksDBOptionsFactory() {
public DBOptions createDBOptions(DBOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
return currentOptions.setMaxOpenFiles(1024);
}
public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
BloomFilter bloomFilter = new BloomFilter();
handlesToClose.add(bloomFilter);
return currentOptions
.setTableFormatConfig(
new BlockBasedTableConfig().setFilter(bloomFilter));
}
});
This is in the main class where we setup in the StreamExecutionEnvironment …
I get ClassLoading errors due to that ...
Caused by: java.lang.LinkageError: loader constraint violation: loader org.apache.flink.util.ChildFirstClassLoader @1169afe1 wants to load class org.rocksdb.ColumnFamilyOptions. A different class with the same name was previously loaded by 'app'. (org.rocksdb.ColumnFamilyOptions is in unnamed module of loader 'app')
What is documented is to change the order to parent-first in the flink-conf.yaml … but then I get different issues for the basic/core Spring Framework classes not being serializable …
Any help will be appreciated.
Thanks,
Sandip
Re: Bloom Filter - RocksDB - LinkageError Classloading
Posted by Sandeep khanzode <sa...@shiftright.ai>.
Hello Stephan, Yun,
Thanks for your insights.
All I have added is this:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-statebackend-rocksdb_2.12</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
No other library explicitly added. I am assuming, as mentioned, is that the flink-dist.jar already contains the relevant classes and the App or parent class loader loads the Rocks DB classes. All other Flink dependencies are packaged as Maven - Provided.
Moving to parent-first gives the Spring Framework serialisation issues … I will take a look at that …
I thought it would be simpler to simply specify Bloom Filters as an option …
Maybe, I will have to remove Spring dependency …
Thanks,
Sandip
> On 05-Aug-2021, at 5:55 PM, Stephan Ewen <se...@apache.org> wrote:
>
> @Yun Tang
>
> Our FRocksDB has the same java package names (org.rocksdb.). Adding 'org.rocksdb' to parent-first patterns ensures it will be loaded only once, and not accidentally multiple times (as Child-first classloading does).
>
> The RocksDB code here is a bit like Flink internal components, which we always force parent-first (everything that starts with "org.apache.fink.").
>
> To use RocksDB from the application jar, I think you would need to remove the RocksDB state backend from the classpath (lib folder), or you get exactly the error reported above.
>
> I cannot think of a downside to add RocksDB to the parent first patterns.
>
> On Thu, Aug 5, 2021 at 10:04 AM Yun Tang <myasuka@live.com <ma...@live.com>> wrote:
> Hi Stephan,
>
> Since we use our own FRocksDB instead of the original RocksDB as dependency, I am not sure whether this problem has relationship with this. From my knowledge, more customers would include Flink classes within the application jar package, and it might cause problems if the client has different flink version with servers.
>
>
> Best,
> Yun Tang
> From: Stephan Ewen <sewen@apache.org <ma...@apache.org>>
> Sent: Wednesday, August 4, 2021 19:10
> To: Yun Tang <myasuka@live.com <ma...@live.com>>
> Cc: Sandeep khanzode <sandeep@shiftright.ai <ma...@shiftright.ai>>; user <user@flink.apache.org <ma...@flink.apache.org>>
> Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading
>
> @Yun Tang Does it make sense to add RocksDB to the "always parent-first options" to avoid these kind of errors when users package apps incorrectly?
> My feeling is that these packaging errors occur very frequently.
>
>
> On Wed, Aug 4, 2021 at 10:41 AM Yun Tang <myasuka@live.com <ma...@live.com>> wrote:
> Hi Sandeep,
>
> Did you include the RocksDB classes in the application jar package? You can unpark your jar package to check whether them existed.
> If so, since RocksDB classes are already included in the flink-dist package, you don't need to include them in your jar package (maybe you explicitly added the dependency of org.rocksdb:rocksdbjni in your pom).
>
> Best
> Yun Tang
> From: Sandeep khanzode <sandeep@shiftright.ai <ma...@shiftright.ai>>
> Sent: Wednesday, August 4, 2021 11:54
> To: user <user@flink.apache.org <ma...@flink.apache.org>>
> Subject: Bloom Filter - RocksDB - LinkageError Classloading
>
> Hello,
>
> I tried to add the bloom filter functionality as mentioned here:
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html <https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html>
>
> rocksDbBackend.setRocksDBOptions(new RocksDBOptionsFactory() {
>
> public DBOptions createDBOptions(DBOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
> return currentOptions.setMaxOpenFiles(1024);
> }
>
> public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
> BloomFilter bloomFilter = new BloomFilter();
> handlesToClose.add(bloomFilter);
>
> return currentOptions
> .setTableFormatConfig(
> new BlockBasedTableConfig().setFilter(bloomFilter));
> }
> });
>
> This is in the main class where we setup in the StreamExecutionEnvironment …
>
> I get ClassLoading errors due to that ...
> Caused by: java.lang.LinkageError: loader constraint violation: loader org.apache.flink.util.ChildFirstClassLoader @1169afe1 wants to load class org.rocksdb.ColumnFamilyOptions. A different class with the same name was previously loaded by 'app'. (org.rocksdb.ColumnFamilyOptions is in unnamed module of loader 'app')
>
>
> What is documented is to change the order to parent-first in the flink-conf.yaml … but then I get different issues for the basic/core Spring Framework classes not being serializable …
>
> Any help will be appreciated.
>
> Thanks,
> Sandip
Re: Bloom Filter - RocksDB - LinkageError Classloading
Posted by Stephan Ewen <se...@apache.org>.
@Yun Tang
Our FRocksDB has the same java package names (org.rocksdb.). Adding
'org.rocksdb' to parent-first patterns ensures it will be loaded only once,
and not accidentally multiple times (as Child-first classloading does).
The RocksDB code here is a bit like Flink internal components, which we
always force parent-first (everything that starts with "org.apache.fink.").
To use RocksDB from the application jar, I think you would need to remove
the RocksDB state backend from the classpath (lib folder), or you get
exactly the error reported above.
I cannot think of a downside to add RocksDB to the parent first patterns.
On Thu, Aug 5, 2021 at 10:04 AM Yun Tang <my...@live.com> wrote:
> Hi Stephan,
>
> Since we use our own FRocksDB instead of the original RocksDB as
> dependency, I am not sure whether this problem has relationship with this.
> From my knowledge, more customers would include Flink classes within the
> application jar package, and it might cause problems if the client has
> different flink version with servers.
>
>
> Best,
> Yun Tang
> ------------------------------
> *From:* Stephan Ewen <se...@apache.org>
> *Sent:* Wednesday, August 4, 2021 19:10
> *To:* Yun Tang <my...@live.com>
> *Cc:* Sandeep khanzode <sa...@shiftright.ai>; user <
> user@flink.apache.org>
> *Subject:* Re: Bloom Filter - RocksDB - LinkageError Classloading
>
> @Yun Tang Does it make sense to add RocksDB to the "always parent-first
> options" to avoid these kind of errors when users package apps incorrectly?
> My feeling is that these packaging errors occur very frequently.
>
>
> On Wed, Aug 4, 2021 at 10:41 AM Yun Tang <my...@live.com> wrote:
>
> Hi Sandeep,
>
> Did you include the RocksDB classes in the application jar package? You
> can unpark your jar package to check whether them existed.
> If so, since RocksDB classes are already included in the flink-dist
> package, you don't need to include them in your jar package (maybe you
> explicitly added the dependency of org.rocksdb:rocksdbjni in your pom).
>
> Best
> Yun Tang
> ------------------------------
> *From:* Sandeep khanzode <sa...@shiftright.ai>
> *Sent:* Wednesday, August 4, 2021 11:54
> *To:* user <us...@flink.apache.org>
> *Subject:* Bloom Filter - RocksDB - LinkageError Classloading
>
> Hello,
>
> I tried to add the bloom filter functionality as mentioned here:
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html
>
> rocksDbBackend.setRocksDBOptions(new RocksDBOptionsFactory() {
>
> public DBOptions createDBOptions(DBOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
> return currentOptions.setMaxOpenFiles(1024);
> }
>
> public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
> BloomFilter bloomFilter = new BloomFilter();
> handlesToClose.add(bloomFilter);
>
> return currentOptions
> .setTableFormatConfig(
> new BlockBasedTableConfig().setFilter(bloomFilter));
> }
> });
>
>
> This is in the main class where we setup in the StreamExecutionEnvironment
> …
>
> I get ClassLoading errors due to that ...
>
> Caused by: java.lang.LinkageError: loader constraint violation: loader org.apache.flink.util.ChildFirstClassLoader @1169afe1 wants to load class org.rocksdb.ColumnFamilyOptions. A different class with the same name was previously loaded by 'app'. (org.rocksdb.ColumnFamilyOptions is in unnamed module of loader 'app')
>
>
>
> What is documented is to change the order to parent-first in the
> flink-conf.yaml … but then I get different issues for the basic/core Spring
> Framework classes not being serializable …
>
> Any help will be appreciated.
>
> Thanks,
> Sandip
>
>
Re: Bloom Filter - RocksDB - LinkageError Classloading
Posted by Yun Tang <my...@live.com>.
Hi Stephan,
Since we use our own FRocksDB instead of the original RocksDB as dependency, I am not sure whether this problem has relationship with this. From my knowledge, more customers would include Flink classes within the application jar package, and it might cause problems if the client has different flink version with servers.
Best,
Yun Tang
________________________________
From: Stephan Ewen <se...@apache.org>
Sent: Wednesday, August 4, 2021 19:10
To: Yun Tang <my...@live.com>
Cc: Sandeep khanzode <sa...@shiftright.ai>; user <us...@flink.apache.org>
Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading
@Yun Tang Does it make sense to add RocksDB to the "always parent-first options" to avoid these kind of errors when users package apps incorrectly?
My feeling is that these packaging errors occur very frequently.
On Wed, Aug 4, 2021 at 10:41 AM Yun Tang <my...@live.com>> wrote:
Hi Sandeep,
Did you include the RocksDB classes in the application jar package? You can unpark your jar package to check whether them existed.
If so, since RocksDB classes are already included in the flink-dist package, you don't need to include them in your jar package (maybe you explicitly added the dependency of org.rocksdb:rocksdbjni in your pom).
Best
Yun Tang
________________________________
From: Sandeep khanzode <sa...@shiftright.ai>>
Sent: Wednesday, August 4, 2021 11:54
To: user <us...@flink.apache.org>>
Subject: Bloom Filter - RocksDB - LinkageError Classloading
Hello,
I tried to add the bloom filter functionality as mentioned here:
https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html
rocksDbBackend.setRocksDBOptions(new RocksDBOptionsFactory() {
public DBOptions createDBOptions(DBOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
return currentOptions.setMaxOpenFiles(1024);
}
public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
BloomFilter bloomFilter = new BloomFilter();
handlesToClose.add(bloomFilter);
return currentOptions
.setTableFormatConfig(
new BlockBasedTableConfig().setFilter(bloomFilter));
}
});
This is in the main class where we setup in the StreamExecutionEnvironment …
I get ClassLoading errors due to that ...
Caused by: java.lang.LinkageError: loader constraint violation: loader org.apache.flink.util.ChildFirstClassLoader @1169afe1 wants to load class org.rocksdb.ColumnFamilyOptions. A different class with the same name was previously loaded by 'app'. (org.rocksdb.ColumnFamilyOptions is in unnamed module of loader 'app')
What is documented is to change the order to parent-first in the flink-conf.yaml … but then I get different issues for the basic/core Spring Framework classes not being serializable …
Any help will be appreciated.
Thanks,
Sandip
Re: Bloom Filter - RocksDB - LinkageError Classloading
Posted by Stephan Ewen <se...@apache.org>.
@Yun Tang Does it make sense to add RocksDB to the "always parent-first
options" to avoid these kind of errors when users package apps incorrectly?
My feeling is that these packaging errors occur very frequently.
On Wed, Aug 4, 2021 at 10:41 AM Yun Tang <my...@live.com> wrote:
> Hi Sandeep,
>
> Did you include the RocksDB classes in the application jar package? You
> can unpark your jar package to check whether them existed.
> If so, since RocksDB classes are already included in the flink-dist
> package, you don't need to include them in your jar package (maybe you
> explicitly added the dependency of org.rocksdb:rocksdbjni in your pom).
>
> Best
> Yun Tang
> ------------------------------
> *From:* Sandeep khanzode <sa...@shiftright.ai>
> *Sent:* Wednesday, August 4, 2021 11:54
> *To:* user <us...@flink.apache.org>
> *Subject:* Bloom Filter - RocksDB - LinkageError Classloading
>
> Hello,
>
> I tried to add the bloom filter functionality as mentioned here:
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html
>
> rocksDbBackend.setRocksDBOptions(new RocksDBOptionsFactory() {
>
> public DBOptions createDBOptions(DBOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
> return currentOptions.setMaxOpenFiles(1024);
> }
>
> public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
> BloomFilter bloomFilter = new BloomFilter();
> handlesToClose.add(bloomFilter);
>
> return currentOptions
> .setTableFormatConfig(
> new BlockBasedTableConfig().setFilter(bloomFilter));
> }
> });
>
>
> This is in the main class where we setup in the StreamExecutionEnvironment
> …
>
> I get ClassLoading errors due to that ...
>
> Caused by: java.lang.LinkageError: loader constraint violation: loader org.apache.flink.util.ChildFirstClassLoader @1169afe1 wants to load class org.rocksdb.ColumnFamilyOptions. A different class with the same name was previously loaded by 'app'. (org.rocksdb.ColumnFamilyOptions is in unnamed module of loader 'app')
>
>
>
> What is documented is to change the order to parent-first in the
> flink-conf.yaml … but then I get different issues for the basic/core Spring
> Framework classes not being serializable …
>
> Any help will be appreciated.
>
> Thanks,
> Sandip
>
Re: Bloom Filter - RocksDB - LinkageError Classloading
Posted by Yun Tang <my...@live.com>.
Hi Sandeep,
Did you include the RocksDB classes in the application jar package? You can unpark your jar package to check whether them existed.
If so, since RocksDB classes are already included in the flink-dist package, you don't need to include them in your jar package (maybe you explicitly added the dependency of org.rocksdb:rocksdbjni in your pom).
Best
Yun Tang
________________________________
From: Sandeep khanzode <sa...@shiftright.ai>
Sent: Wednesday, August 4, 2021 11:54
To: user <us...@flink.apache.org>
Subject: Bloom Filter - RocksDB - LinkageError Classloading
Hello,
I tried to add the bloom filter functionality as mentioned here:
https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html
rocksDbBackend.setRocksDBOptions(new RocksDBOptionsFactory() {
public DBOptions createDBOptions(DBOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
return currentOptions.setMaxOpenFiles(1024);
}
public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions currentOptions, Collection<AutoCloseable> handlesToClose) {
BloomFilter bloomFilter = new BloomFilter();
handlesToClose.add(bloomFilter);
return currentOptions
.setTableFormatConfig(
new BlockBasedTableConfig().setFilter(bloomFilter));
}
});
This is in the main class where we setup in the StreamExecutionEnvironment …
I get ClassLoading errors due to that ...
Caused by: java.lang.LinkageError: loader constraint violation: loader org.apache.flink.util.ChildFirstClassLoader @1169afe1 wants to load class org.rocksdb.ColumnFamilyOptions. A different class with the same name was previously loaded by 'app'. (org.rocksdb.ColumnFamilyOptions is in unnamed module of loader 'app')
What is documented is to change the order to parent-first in the flink-conf.yaml … but then I get different issues for the basic/core Spring Framework classes not being serializable …
Any help will be appreciated.
Thanks,
Sandip