You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by lk gen <lk...@gmail.com> on 2018/11/27 17:20:07 UTC

problems in Kafka unit testing trunk

When running ./gradlew test
on a centos machine with gradle and java set
In the trunk version from today

There are errors about too many files open of the form
"
kafka.admin.DeleteTopicTest > testDeletingPartiallyDeletedTopic FAILED
    org.apache.kafka.common.KafkaException: java.io.IOException: Too many
open files
        at
org.apache.kafka.common.network.Selector.<init>(Selector.java:160)
        at
org.apache.kafka.common.network.Selector.<init>(Selector.java:212)
        at
org.apache.kafka.common.network.Selector.<init>(Selector.java:225)
        at
kafka.coordinator.transaction.TransactionMarkerChannelManager$.apply(TransactionMarkerChannelManager.scala:66)
        at
kafka.coordinator.transaction.TransactionCoordinator$.apply(TransactionCoordinator.scala:62)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:279)
        at kafka.utils.TestUtils$.createServer(TestUtils.scala:135)
        at
kafka.admin.DeleteTopicTest.$anonfun$createTestTopicAndCluster$2(DeleteTopicTest.scala:372)
        at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
        at scala.collection.Iterator.foreach(Iterator.scala:937)
        at scala.collection.Iterator.foreach$(Iterator.scala:937)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
        at scala.collection.IterableLike.foreach(IterableLike.scala:70)
        at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at scala.collection.TraversableLike.map(TraversableLike.scala:233)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at
kafka.admin.DeleteTopicTest.createTestTopicAndCluster(DeleteTopicTest.scala:372)
        at
kafka.admin.DeleteTopicTest.createTestTopicAndCluster(DeleteTopicTest.scala:366)
        at
kafka.admin.DeleteTopicTest.testDeletingPartiallyDeletedTopic(DeleteTopicTest.scala:418)

        Caused by:
        java.io.IOException: Too many open files
            at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)
            at
sun.nio.ch.EPollArrayWrapper.<init>(EPollArrayWrapper.java:130)
            at
sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:69)
            at
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
            at java.nio.channels.Selector.open(Selector.java:227)
            at
org.apache.kafka.common.network.Selector.<init>(Selector.java:158)
            ... 20 more

"

Is the environment I am using for gradle test is invalid ? are there
special settings required ?

Re: problems in Kafka unit testing trunk

Posted by lk gen <lk...@gmail.com>.

ulimit shows
$ ulimit -n
1024

is it too small for Kafka ?




On Wed, Nov 28, 2018 at 6:18 AM Dhruvil Shah <dh...@confluent.io> wrote:

> The unit test itself does not seem to use too many files. What is the
> output for `ulimit -n` on your system? Running `lsof` might also be helpful
> to determine how many open files you have while Kafka is not running.
>
> - Dhruvil
>
> On Tue, Nov 27, 2018 at 9:20 AM lk gen <lk...@gmail.com> wrote:
>
> > When running ./gradlew test
> > on a centos machine with gradle and java set
> > In the trunk version from today
> >
> > There are errors about too many files open of the form
> > "
> > kafka.admin.DeleteTopicTest > testDeletingPartiallyDeletedTopic FAILED
> >     org.apache.kafka.common.KafkaException: java.io.IOException: Too many
> > open files
> >         at
> > org.apache.kafka.common.network.Selector.<init>(Selector.java:160)
> >         at
> > org.apache.kafka.common.network.Selector.<init>(Selector.java:212)
> >         at
> > org.apache.kafka.common.network.Selector.<init>(Selector.java:225)
> >         at
> >
> >
> kafka.coordinator.transaction.TransactionMarkerChannelManager$.apply(TransactionMarkerChannelManager.scala:66)
> >         at
> >
> >
> kafka.coordinator.transaction.TransactionCoordinator$.apply(TransactionCoordinator.scala:62)
> >         at kafka.server.KafkaServer.startup(KafkaServer.scala:279)
> >         at kafka.utils.TestUtils$.createServer(TestUtils.scala:135)
> >         at
> >
> >
> kafka.admin.DeleteTopicTest.$anonfun$createTestTopicAndCluster$2(DeleteTopicTest.scala:372)
> >         at
> >
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
> >         at scala.collection.Iterator.foreach(Iterator.scala:937)
> >         at scala.collection.Iterator.foreach$(Iterator.scala:937)
> >         at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
> >         at scala.collection.IterableLike.foreach(IterableLike.scala:70)
> >         at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
> >         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> >         at
> scala.collection.TraversableLike.map(TraversableLike.scala:233)
> >         at
> scala.collection.TraversableLike.map$(TraversableLike.scala:226)
> >         at
> scala.collection.AbstractTraversable.map(Traversable.scala:104)
> >         at
> >
> >
> kafka.admin.DeleteTopicTest.createTestTopicAndCluster(DeleteTopicTest.scala:372)
> >         at
> >
> >
> kafka.admin.DeleteTopicTest.createTestTopicAndCluster(DeleteTopicTest.scala:366)
> >         at
> >
> >
> kafka.admin.DeleteTopicTest.testDeletingPartiallyDeletedTopic(DeleteTopicTest.scala:418)
> >
> >         Caused by:
> >         java.io.IOException: Too many open files
> >             at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)
> >             at
> > sun.nio.ch.EPollArrayWrapper.<init>(EPollArrayWrapper.java:130)
> >             at
> > sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:69)
> >             at
> > sun.nio.ch
> > .EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
> >             at java.nio.channels.Selector.open(Selector.java:227)
> >             at
> > org.apache.kafka.common.network.Selector.<init>(Selector.java:158)
> >             ... 20 more
> >
> > "
> >
> > Is the environment I am using for gradle test is invalid ? are there
> > special settings required ?
> >
>

Re: problems in Kafka unit testing trunk

Posted by Dhruvil Shah <dh...@confluent.io>.

The unit test itself does not seem to use too many files. What is the
output for `ulimit -n` on your system? Running `lsof` might also be helpful
to determine how many open files you have while Kafka is not running.

- Dhruvil

On Tue, Nov 27, 2018 at 9:20 AM lk gen <lk...@gmail.com> wrote:

> When running ./gradlew test
> on a centos machine with gradle and java set
> In the trunk version from today
>
> There are errors about too many files open of the form
> "
> kafka.admin.DeleteTopicTest > testDeletingPartiallyDeletedTopic FAILED
>     org.apache.kafka.common.KafkaException: java.io.IOException: Too many
> open files
>         at
> org.apache.kafka.common.network.Selector.<init>(Selector.java:160)
>         at
> org.apache.kafka.common.network.Selector.<init>(Selector.java:212)
>         at
> org.apache.kafka.common.network.Selector.<init>(Selector.java:225)
>         at
>
> kafka.coordinator.transaction.TransactionMarkerChannelManager$.apply(TransactionMarkerChannelManager.scala:66)
>         at
>
> kafka.coordinator.transaction.TransactionCoordinator$.apply(TransactionCoordinator.scala:62)
>         at kafka.server.KafkaServer.startup(KafkaServer.scala:279)
>         at kafka.utils.TestUtils$.createServer(TestUtils.scala:135)
>         at
>
> kafka.admin.DeleteTopicTest.$anonfun$createTestTopicAndCluster$2(DeleteTopicTest.scala:372)
>         at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
>         at scala.collection.Iterator.foreach(Iterator.scala:937)
>         at scala.collection.Iterator.foreach$(Iterator.scala:937)
>         at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
>         at scala.collection.IterableLike.foreach(IterableLike.scala:70)
>         at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
>         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>         at scala.collection.TraversableLike.map(TraversableLike.scala:233)
>         at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
>         at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>         at
>
> kafka.admin.DeleteTopicTest.createTestTopicAndCluster(DeleteTopicTest.scala:372)
>         at
>
> kafka.admin.DeleteTopicTest.createTestTopicAndCluster(DeleteTopicTest.scala:366)
>         at
>
> kafka.admin.DeleteTopicTest.testDeletingPartiallyDeletedTopic(DeleteTopicTest.scala:418)
>
>         Caused by:
>         java.io.IOException: Too many open files
>             at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)
>             at
> sun.nio.ch.EPollArrayWrapper.<init>(EPollArrayWrapper.java:130)
>             at
> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:69)
>             at
> sun.nio.ch
> .EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
>             at java.nio.channels.Selector.open(Selector.java:227)
>             at
> org.apache.kafka.common.network.Selector.<init>(Selector.java:158)
>             ... 20 more
>
> "
>
> Is the environment I am using for gradle test is invalid ? are there
> special settings required ?
>