You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kevin Niemann <Ke...@expeditors.com> on 2016/04/12 17:28:19 UTC
RocksDB on Windows
I've built Kafka from trunk 0.10 intending to build a streaming application and I'm wondering it if supports Windows. I tried increasing the Maven dependency to RocksDB 4.2.0.
Jay's post mentions Kafka support multiple state stores. Is it documented how to configure these or do you recommend running my app in Linux?
public static String getJniLibraryName(final String name) {
if (isUnix()) {
final String arch = (is64Bit()) ? "64" : "32";
return String.format("%sjni-linux%s", name, arch);
} else if (isMac()) {
return String.format("%sjni-osx", name);
}
throw new UnsupportedOperationException();
}
Exception in thread "StreamThread-1" java.lang.ExceptionInInitializerError
at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:47)
at org.rocksdb.RocksDB.<clinit>(RocksDB.java:23)
at org.rocksdb.Options.<clinit>(Options.java:21)
at org.apache.kafka.streams.state.internals.RocksDBStore.<init>(RocksDBStore.java:113)
at org.apache.kafka.streams.state.internals.RocksDBStore.<init>(RocksDBStore.java:99)
at org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreSupplier.get(RocksDBKeyValueStoreSupplier.java:56)
at org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:77)
at org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:115)
at org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:579)
at org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:606)
at org.apache.kafka.streams.processor.internals.StreamThread.access$000(StreamThread.java:71)
at org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:126)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:205)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:226)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:221)
at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
at org.apache.kafka.clients.consumer.internals.RequestFuture$2.onSuccess(RequestFuture.java:182)
at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupRequestHandler.handle(AbstractCoordinator.java:430)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupRequestHandler.handle(AbstractCoordinator.java:416)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:673)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:652)
at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:397)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:279)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:338)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:212)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:191)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:161)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:237)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:323)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:900)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:862)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:325)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:248)
Caused by: java.lang.UnsupportedOperationException
at org.rocksdb.util.Environment.getJniLibraryName(Environment.java:40)
at org.rocksdb.NativeLibraryLoader.<clinit>(NativeLibraryLoader.java:19)
... 39 more
Thanks,
Kevin
Re: RocksDB on Windows
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Kevin,
You can use the Processor API to create multiple state stores and associate
these stores to your processors, some guidance can be found here:
http://kafka.apache.org/0100/documentation.html#streams_processor
As for your specific case, unfortunately I cannot tell much from your code
snippet and the stack trace, if you could update your program sketch
defining the topology I can help you defining what king of apis you may
want to use.
Guozhang
On Tue, Apr 12, 2016 at 8:28 AM, Kevin Niemann <Kevin.Niemann@expeditors.com
> wrote:
> I've built Kafka from trunk 0.10 intending to build a streaming
> application and I'm wondering it if supports Windows. I tried increasing
> the Maven dependency to RocksDB 4.2.0.
>
> Jay's post mentions Kafka support multiple state stores. Is it documented
> how to configure these or do you recommend running my app in Linux?
>
> public static String getJniLibraryName(final String name) {
> if (isUnix()) {
> final String arch = (is64Bit()) ? "64" : "32";
> return String.format("%sjni-linux%s", name, arch);
> } else if (isMac()) {
> return String.format("%sjni-osx", name);
> }
> throw new UnsupportedOperationException();
> }
>
> Exception in thread "StreamThread-1" java.lang.ExceptionInInitializerError
> at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:47)
> at org.rocksdb.RocksDB.<clinit>(RocksDB.java:23)
> at org.rocksdb.Options.<clinit>(Options.java:21)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.<init>(RocksDBStore.java:113)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.<init>(RocksDBStore.java:99)
> at
> org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreSupplier.get(RocksDBKeyValueStoreSupplier.java:56)
> at
> org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:77)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:115)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:579)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:606)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.access$000(StreamThread.java:71)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:126)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:205)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:226)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:221)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture$2.onSuccess(RequestFuture.java:182)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupRequestHandler.handle(AbstractCoordinator.java:430)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupRequestHandler.handle(AbstractCoordinator.java:416)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:673)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:652)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:397)
> at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:279)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:338)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:212)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:191)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:161)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:237)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:323)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:900)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:862)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:325)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:248)
> Caused by: java.lang.UnsupportedOperationException
> at
> org.rocksdb.util.Environment.getJniLibraryName(Environment.java:40)
> at
> org.rocksdb.NativeLibraryLoader.<clinit>(NativeLibraryLoader.java:19)
> ... 39 more
>
> Thanks,
> Kevin
>
--
-- Guozhang