Software Updater

Merge lp:~jibel/update-manager/AutoUpgradeTester-portlocking into lp:update-manager

  1. AutoUpgradeTester-portlocking
  2. Merge into main
Proposed by Jean-Baptiste Lallement
Status: Merged
Merged at revision: 2326
Proposed branch: lp:~jibel/update-manager/AutoUpgradeTester-portlocking
Merge into: lp:update-manager
Diff against target: 125 lines (+57/-11)
2 files modified
AutoUpgradeTester/UpgradeTestBackendQemu.py (+54/-10)
AutoUpgradeTester/post_upgrade_tests/debconf_test.py (+3/-1)
To merge this branch: bzr merge lp:~jibel/update-manager/AutoUpgradeTester-portlocking
Reviewer Review Type Date Requested Status
Michael Vogt (community) Approve
Review via email: mp+90695@code.launchpad.net

Description of the change

This patch provides the ability to start auto-upgrade-tester runs in parallel.
SSH and VNC ports were not seen as allocated until services were really started by KVM. Now, lock files are used to reserve port for later use by KVM. Tested with Jenkins and 2 executors on each node.

For reference, I used fcntl instead of python-lockfile:
- To avoid a new dependency
- python-lockfile doesn't support locking 2 files from the same process. (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=632857)

To post a comment you must log in.
Revision history for this message
Michael Vogt (mvo) wrote :

Thanks a lot for doing this work!

I merged this with some small modifications. E.g. in getFreePort I raise a exception instead of sys.exit() to allow the consumer of the class to decide what to do.

Cheers,
 Michael

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'AutoUpgradeTester/UpgradeTestBackendQemu.py'
2--- AutoUpgradeTester/UpgradeTestBackendQemu.py 2012-01-25 22:51:09 +0000
3+++ AutoUpgradeTester/UpgradeTestBackendQemu.py 2012-01-30 14:29:04 +0000
4@@ -14,6 +14,7 @@
5 import tempfile
6 import atexit
7 import apt_pkg
8+import fcntl
9
10 from DistUpgrade.utils import is_port_already_listening
11
12@@ -108,8 +109,10 @@
13 self.image = os.path.join(imagedir, "test-image.%s" % self.profilename)
14 # make ssh login possible (localhost 54321) available
15 ssh_port = int(self.config.getWithDefault("KVM","SshPort","54321"))
16- while is_port_already_listening(ssh_port):
17- ssh_port += 1
18+ (self.ssh_lock, ssh_port) = self.getFreePort(port_base = ssh_port)
19+ if not self.ssh_lock:
20+ print "ERROR: Couldn't allocate SSH port. Exiting!"
21+ sys.exit(1)
22 self.ssh_port = str(ssh_port)
23 print "using ssh port: %s" % self.ssh_port
24 self.ssh_hostname = "localhost"
25@@ -117,12 +120,14 @@
26 self.qemu_options.append("tcp:%s::22" % self.ssh_port)
27 # vnc port/display
28 VNC_BASE_PORT = 5900
29- vncport = int(self.config.getWithDefault("KVM","VncNum", "0"))
30- while is_port_already_listening(VNC_BASE_PORT + vncport):
31- vncport += 1
32+ vncport = int(self.config.getWithDefault("KVM","VncNum", "0")) + VNC_BASE_PORT
33+ (self.vnc_lock, vncport) = self.getFreePort(port_base = vncport)
34+ if not self.vnc_lock:
35+ print "ERROR: Couldn't allocate VNC port. Exiting!"
36+ sys.exit(1)
37 print "using VncNum: %s" % vncport
38 self.qemu_options.append("-vnc")
39- self.qemu_options.append("localhost:%s" % str(vncport))
40+ self.qemu_options.append("localhost:%s" % str(vncport - VNC_BASE_PORT))
41
42 # make the memory configurable
43 mem = self.config.getWithDefault("KVM","VirtualRam","1536")
44@@ -136,8 +141,19 @@
45 # register exit handler to ensure that we quit kvm on exit
46 atexit.register(self.stop)
47
48+ def __del__(self):
49+ """
50+ Destructor
51+ Clean-up lockfiles
52+ """
53+ for lock in (self.ssh_lock, self.vnc_lock):
54+ lockpath = lock.name
55+ print "Releasing lock: %s" % lockpath
56+ lock.close()
57+ os.unlink(lockpath)
58+
59 def genDiff(self):
60- """
61+ """
62 generate a diff that compares a fresh install to a upgrade.
63 ideally that should be empty
64 Ensure that we always run this *after* the regular upgrade was
65@@ -519,14 +535,42 @@
66 print "Shuting down the VM"
67 self.stop()
68 return (ret == 0)
69-
70+
71+ def getFreePort(self, port_base=1025, prefix='auto-upgrade-tester'):
72+ """ Find a free port and lock it when found
73+ :param port_base: Base port number.
74+ :param prefix: Prefix name for the lock
75+ :return: (lockfile, portnumber)
76+ """
77+
78+ lockdir = '/var/lock'
79+
80+ for port_inc in range(0, 100):
81+ port_num = port_base + port_inc
82+ if is_port_already_listening(port_num):
83+ print "Port %d already in use. Skipping!" % port_num
84+ continue
85+
86+ lockfilepath = os.path.join(lockdir, '%s.%d.lock' % (prefix, port_num))
87+ if not os.path.exists(lockfilepath):
88+ open(lockfilepath, 'w').close()
89+ lock = open(lockfilepath, 'r+')
90+ try:
91+ fcntl.flock(lock, fcntl.LOCK_EX|fcntl.LOCK_NB)
92+ return (lock, port_num)
93+ except IOError:
94+ print "Port %d already locked. Skipping!" % port_num
95+ lock.close()
96+
97+ print "No free port found. Aborting!"
98+ return (None, None)
99
100 if __name__ == "__main__":
101-
102+
103 # FIXME: very rough proof of conecpt, unify with the chroot
104 # and automatic-upgrade code
105 # see also /usr/sbin/qemu-make-debian-root
106-
107+
108 qemu = UpgradeTestBackendQemu(sys.argv[1],".")
109 #qemu.bootstrap()
110 #qemu.start()
111
112=== modified file 'AutoUpgradeTester/post_upgrade_tests/debconf_test.py'
113--- AutoUpgradeTester/post_upgrade_tests/debconf_test.py 2012-01-18 14:45:04 +0000
114+++ AutoUpgradeTester/post_upgrade_tests/debconf_test.py 2012-01-30 14:29:04 +0000
115@@ -12,7 +12,9 @@
116
117 # Prompts in this list won't generate a test failure
118 # i.e WHITELIST = ['libraries/restart-without-asking']
119-WHITELIST = []
120+WHITELIST = [
121+ 'glibc/restart-services',
122+ 'libraries/restart-without-asking' ]
123
124 def run_test(logfile, resultdir):
125 """ Run the test and slice debconf log

Subscribers

People subscribed via source and target branches

to status/vote changes: