You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Grant Henke <gh...@cloudera.com> on 2015/07/19 17:24:17 UTC
Kafka Unit Test Failures on a Mac
When running all Kafka tests I had been getting failures most every time.
Usually in the SocketServerTest class. However, when I would run individual
tests, there were no failures. After a bit of digging I found this is due
to the small default open files limit in Mac Yosemite. I am positing how to
increase the limit here in case anyone else has been running into the
issue. Let me know if this helped you too. If it is fairly common we can
put something on the wiki.
*Adjusting Open File Limits in Yosemite:*
Note: You can choose your own limits as appropriate
1. Write the following xml to /Library/LaunchDaemons/limit.maxfiles.plist:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "
http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>limit.maxfiles</string>
<key>ProgramArguments</key>
<array>
<string>launchctl</string>
<string>limit</string>
<string>maxfiles</string>
<string>65536</string>
<string>65536</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>ServiceIPC</key>
<false/>
</dict>
</plist>
2. Then write the following to /Library/LaunchDaemons/limit.maxproc.plist:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "
http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>limit.maxproc</string>
<key>ProgramArguments</key>
<array>
<string>launchctl</string>
<string>limit</string>
<string>maxproc</string>
<string>2048</string>
<string>2048</string>
</array>
<key>RunAtLoad</key>
<true />
<key>ServiceIPC</key>
<false />
</dict>
</plist>
3. Add the following to your bashrc or bashprofile:
ulimit -n 65536
ulimit -u 2048
4. Restart your computer. After restart validate settings by executing:
launchctl limit
*Adjusting Open File Limits in Older Versions of OS X:*
Note: You can choose your own limits as appropriate
1. Add the following command to /etc/launchd.conf:
limit maxfiles 32768 65536
2. Restart your computer. After restart validate settings by executing:
launchctl limit
Re: Kafka Unit Test Failures on a Mac
Posted by Ismael Juma <is...@juma.me.uk>.
On Mon, Jul 20, 2015 at 10:45 PM, Grant Henke <gh...@cloudera.com> wrote:
> In one run of the core tests I found the following:
>
> - 4584 regular files (REG)
> - 376 .jar files
> - Not much one can/should do here. Many are from gradle itself.
> - 2392 kafka .log files
> - why are these being leaked?
> - after a single test no file handles should remain
> - 1162 kafka .log.deleted files
> - why are these being leaked?
>
Some of those are probably due to:
https://issues.apache.org/jira/browse/KAFKA-1782
Ismael
Re: Kafka Unit Test Failures on a Mac
Posted by Grant Henke <gh...@cloudera.com>.
Thanks Ismael!
I agree clear failures or no failures is optimal. I did some hacky analysis
of the open files by running the tests and utilizing the lsof command.
In one run of the core tests I found the following:
- 4584 regular files (REG)
- 376 .jar files
- Not much one can/should do here. Many are from gradle itself.
- 2392 kafka .log files
- why are these being leaked?
- after a single test no file handles should remain
- 1162 kafka .log.deleted files
- why are these being leaked?
- 469 kafka .index files
- This is due to Java's handling of MappedByteBuffer
- A mapped byte buffer and the file mapping that it represents
remain valid until the buffer itself is garbage-collected.
- http://bugs.java.com/view_bug.do?bug_id=4724038
-
http://stackoverflow.com/questions/2972986/how-to-unmap-a-file-from-memory-mapped-using-filechannel-in-java
- Perhaps setting mmap to null when kafka.log.OffsetIndex.close
is called would help ensure this gets GC'd asap.
- 943 of types PIPE & KQUEUE
- 629 PIPE
- 314 KQUEUE
- should do some analysis sometime
- 47 of other types (TCP, unix, IPv6, ...)
On Sun, Jul 19, 2015 at 3:16 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Hello Grant,
>
> Thanks for figuring this out. I have also run into this issue when running
> the tests on OS X Yosemite.
>
> Ideally the tests would fail in a way that would make it clear what the
> issue is. That may be complicated, so we should at least document it as you
> suggest.
>
> I'll let you know if the issues goes away for me too with this change.
>
> Best,
> Ismael
>
> On Sun, Jul 19, 2015 at 4:24 PM, Grant Henke <gh...@cloudera.com> wrote:
>
> > When running all Kafka tests I had been getting failures most every time.
> > Usually in the SocketServerTest class. However, when I would run
> individual
> > tests, there were no failures. After a bit of digging I found this is due
> > to the small default open files limit in Mac Yosemite. I am positing how
> to
> > increase the limit here in case anyone else has been running into the
> > issue. Let me know if this helped you too. If it is fairly common we can
> > put something on the wiki.
> >
> > *Adjusting Open File Limits in Yosemite:*
> > Note: You can choose your own limits as appropriate
> >
> > 1. Write the following xml to
> /Library/LaunchDaemons/limit.maxfiles.plist:
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> >
> > <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "
> > http://www.apple.com/DTDs/PropertyList-1.0.dtd">
> >
> > <plist version="1.0">
> >
> > <dict>
> >
> > <key>Label</key>
> >
> > <string>limit.maxfiles</string>
> >
> > <key>ProgramArguments</key>
> >
> > <array>
> >
> > <string>launchctl</string>
> >
> > <string>limit</string>
> >
> > <string>maxfiles</string>
> >
> > <string>65536</string>
> >
> > <string>65536</string>
> >
> > </array>
> >
> > <key>RunAtLoad</key>
> >
> > <true/>
> >
> > <key>ServiceIPC</key>
> >
> > <false/>
> >
> > </dict>
> >
> > </plist>
> >
> >
> > 2. Then write the following to
> /Library/LaunchDaemons/limit.maxproc.plist:
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "
> > http://www.apple.com/DTDs/PropertyList-1.0.dtd">
> > <plist version="1.0">
> > <dict>
> > <key>Label</key>
> > <string>limit.maxproc</string>
> > <key>ProgramArguments</key>
> > <array>
> > <string>launchctl</string>
> > <string>limit</string>
> > <string>maxproc</string>
> > <string>2048</string>
> > <string>2048</string>
> > </array>
> > <key>RunAtLoad</key>
> > <true />
> > <key>ServiceIPC</key>
> > <false />
> > </dict>
> > </plist>
> >
> >
> > 3. Add the following to your bashrc or bashprofile:
> >
> > ulimit -n 65536
> >
> > ulimit -u 2048
> >
> >
> > 4. Restart your computer. After restart validate settings by executing:
> >
> > launchctl limit
> >
> >
> >
> > *Adjusting Open File Limits in Older Versions of OS X:*
> > Note: You can choose your own limits as appropriate
> >
> > 1. Add the following command to /etc/launchd.conf:
> >
> > limit maxfiles 32768 65536
> >
> >
> > 2. Restart your computer. After restart validate settings by executing:
> >
> > launchctl limit
> >
>
--
Grant Henke
Solutions Consultant | Cloudera
ghenke@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Re: Kafka Unit Test Failures on a Mac
Posted by Ismael Juma <is...@juma.me.uk>.
Hello Grant,
Thanks for figuring this out. I have also run into this issue when running
the tests on OS X Yosemite.
Ideally the tests would fail in a way that would make it clear what the
issue is. That may be complicated, so we should at least document it as you
suggest.
I'll let you know if the issues goes away for me too with this change.
Best,
Ismael
On Sun, Jul 19, 2015 at 4:24 PM, Grant Henke <gh...@cloudera.com> wrote:
> When running all Kafka tests I had been getting failures most every time.
> Usually in the SocketServerTest class. However, when I would run individual
> tests, there were no failures. After a bit of digging I found this is due
> to the small default open files limit in Mac Yosemite. I am positing how to
> increase the limit here in case anyone else has been running into the
> issue. Let me know if this helped you too. If it is fairly common we can
> put something on the wiki.
>
> *Adjusting Open File Limits in Yosemite:*
> Note: You can choose your own limits as appropriate
>
> 1. Write the following xml to /Library/LaunchDaemons/limit.maxfiles.plist:
>
> <?xml version="1.0" encoding="UTF-8"?>
>
> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "
> http://www.apple.com/DTDs/PropertyList-1.0.dtd">
>
> <plist version="1.0">
>
> <dict>
>
> <key>Label</key>
>
> <string>limit.maxfiles</string>
>
> <key>ProgramArguments</key>
>
> <array>
>
> <string>launchctl</string>
>
> <string>limit</string>
>
> <string>maxfiles</string>
>
> <string>65536</string>
>
> <string>65536</string>
>
> </array>
>
> <key>RunAtLoad</key>
>
> <true/>
>
> <key>ServiceIPC</key>
>
> <false/>
>
> </dict>
>
> </plist>
>
>
> 2. Then write the following to /Library/LaunchDaemons/limit.maxproc.plist:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "
> http://www.apple.com/DTDs/PropertyList-1.0.dtd">
> <plist version="1.0">
> <dict>
> <key>Label</key>
> <string>limit.maxproc</string>
> <key>ProgramArguments</key>
> <array>
> <string>launchctl</string>
> <string>limit</string>
> <string>maxproc</string>
> <string>2048</string>
> <string>2048</string>
> </array>
> <key>RunAtLoad</key>
> <true />
> <key>ServiceIPC</key>
> <false />
> </dict>
> </plist>
>
>
> 3. Add the following to your bashrc or bashprofile:
>
> ulimit -n 65536
>
> ulimit -u 2048
>
>
> 4. Restart your computer. After restart validate settings by executing:
>
> launchctl limit
>
>
>
> *Adjusting Open File Limits in Older Versions of OS X:*
> Note: You can choose your own limits as appropriate
>
> 1. Add the following command to /etc/launchd.conf:
>
> limit maxfiles 32768 65536
>
>
> 2. Restart your computer. After restart validate settings by executing:
>
> launchctl limit
>