juju cannot connect to zookeeper - too many connections (max 10)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
juju (Ubuntu) |
New
|
Medium
|
Unassigned |
Bug Description
My juju cluster running on precise had severe connection issues to zookeeper. Looking at zookeeper logs I can see that my juju nodes cannot connect as they already have pending connections.
Zookeeper log is full of lines like (quick count gives me around 30,000)
2012-11-08 06:46:48,777 - WARN [NIOServerCxn.
There are also session expiration messages (around 1000)
2012-11-08 06:46:58,000 - INFO [SessionTracker
And exceptions (about 4000)
2012-11-08 06:47:05,663 - ERROR [SyncThread:
java.nio.
at sun.nio.
at sun.nio.
at org.apache.
at org.apache.
at org.apache.
at org.apache.
ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: juju 0.5+bzr531-
ProcVersionSign
Uname: Linux 3.2.0-32-virtual x86_64
ApportVersion: 2.0.1-0ubuntu14
Architecture: amd64
Date: Tue Nov 13 11:11:30 2012
Ec2AMI: ami-000000bf
Ec2AMIManifest: FIXME
Ec2Availability
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitec
ProcEnviron:
TERM=xterm-
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: juju
UpgradeStatus: No upgrade log present (probably fresh install)
Zookeeper includes a default limit per client of 10 connections; this is to ensure that a single rogue client can't kill a zookeeper cluster.
This points to something else bad happening in your environment.
If more than ten connections per client are required for juju, then maxClientCnxns can be specified in the zookeeper config file on the bootstrap node - although I don't think this should be done.