Duplicity

Merge lp:~ed.so/duplicity/webdav.lftp.ssl-overhaul into lp:~duplicity-team/duplicity/0.7-series

  1. webdav.lftp.ssl-overhaul
  2. Merge into 0.7-series
Proposed by edso
Status: Merged
Merged at revision: 1196
Proposed branch: lp:~ed.so/duplicity/webdav.lftp.ssl-overhaul
Merge into: lp:~duplicity-team/duplicity/0.7-series
Diff against target: 404 lines (+95/-65)
6 files modified
bin/duplicity.1 (+20/-4)
duplicity/backend.py (+2/-0)
duplicity/backends/lftpbackend.py (+22/-21)
duplicity/backends/webdavbackend.py (+47/-37)
duplicity/commandline.py (+2/-2)
duplicity/globals.py (+2/-1)
To merge this branch: bzr merge lp:~ed.so/duplicity/webdav.lftp.ssl-overhaul
Reviewer Review Type Date Requested Status
duplicity-team Pending
Review via email: mp+287654@code.launchpad.net

Description of the change

duplicity.1, commandline.py, globals.py
- added --ssl-cacert-path parameter
backend.py
- make sure url path component is properly url decoded,
  in case it contains special chars (eg. @ or space)
lftpbackend.py
- quote _all_ cmd line params
- added missing lftp+ftpes protocol
- fix empty list result when chdir failed silently
- added ssl_cacert_path support
webdavbackend.py
- add ssl default context support for python 2.7.9+
  (using system certs eg. in /etc/ssl/certs)
- added ssl_cacert_path support for python 2.7.9+
- gettext wrapped all log messages
- minor refinements

To post a comment you must log in.
lp:~ed.so/duplicity/webdav.lftp.ssl-overhaul updated
1197. By ed.so

path may be unset

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'bin/duplicity.1'
2--- bin/duplicity.1 2016-02-18 18:22:35 +0000
3+++ bin/duplicity.1 2016-03-01 16:25:36 +0000
4@@ -854,15 +854,23 @@
5
6 .TP
7 .BI "--ssl-cacert-file " file
8-.B (only webdav backend)
9+.B (only webdav & lftp backend)
10 Provide a cacert file for ssl certificate verification.
11 .br
12 See also
13 .BR "A NOTE ON SSL CERTIFICATE VERIFICATION" .
14
15 .TP
16+.BI "--ssl-cacert-path " path/to/certs/
17+.B (only webdav backend and python 2.7.9+ OR lftp+webdavs and a recent lftp)
18+Provide a path to a folder containing cacert files for ssl certificate verification.
19+.br
20+See also
21+.BR "A NOTE ON SSL CERTIFICATE VERIFICATION" .
22+
23+.TP
24 .BI --ssl-no-check-certificate
25-.B (only webdav backend)
26+.B (only webdav & lftp backend)
27 Disable ssl certificate verification.
28 .br
29 See also
30@@ -1873,8 +1881,16 @@
31 an sftp service running on the backend server, which is sometimes not an option.
32
33 .SH A NOTE ON SSL CERTIFICATE VERIFICATION
34-Certificate verification as implemented right now [01.2013] only in the webdav backend needs a file
35-based database of certification authority certificates (cacert file). It has to be a
36+Certificate verification as implemented right now [02.2016] only in the webdav
37+and lftp backends. older pythons 2.7.8- and older lftp binaries need a file
38+based database of certification authority certificates (cacert file).
39+.br
40+Newer python 2.7.9+ and recent lftp versions however support the system default
41+certificates (usually in /etc/ssl/certs) and also giving an alternative ca cert
42+folder via
43+.BR --ssl-cacert-path .
44+.PP
45+The cacert file has to be a
46 .B PEM
47 formatted text file as currently provided by the
48 .B CURL
49
50=== modified file 'duplicity/backend.py'
51--- duplicity/backend.py 2016-01-24 17:56:38 +0000
52+++ duplicity/backend.py 2016-03-01 16:25:36 +0000
53@@ -266,6 +266,8 @@
54
55 try:
56 self.path = pu.path
57+ if self.path:
58+ self.path = urllib.unquote(self.path)
59 except Exception:
60 raise InvalidBackendURL("Syntax error (path) in: %s" % url_string)
61
62
63=== modified file 'duplicity/backends/lftpbackend.py'
64--- duplicity/backends/lftpbackend.py 2016-01-24 17:30:02 +0000
65+++ duplicity/backends/lftpbackend.py 2016-03-01 16:25:36 +0000
66@@ -95,24 +95,21 @@
67 cacert_candidates = ["~/.duplicity/cacert.pem",
68 "~/duplicity_cacert.pem",
69 "/etc/duplicity/cacert.pem"]
70- #
71+ # look for a default cacert file
72 if not self.cacert_file:
73 for path in cacert_candidates:
74 path = os.path.expanduser(path)
75 if (os.path.isfile(path)):
76 self.cacert_file = path
77 break
78- # still no cacert file, inform user
79- if not self.cacert_file:
80- raise duplicity.errors.FatalBackendException("""For certificate verification a cacert database file is needed in one of these locations: %s
81-Hints:
82- Consult the man page, chapter 'SSL Certificate Verification'.
83- Consider using the options --ssl-cacert-file, --ssl-no-check-certificate .""" % ", ".join(cacert_candidates))
84
85+ # save config into a reusable temp file
86 self.tempfile, self.tempname = tempdir.default().mkstemp()
87 os.write(self.tempfile, "set ssl:verify-certificate " + ("false" if globals.ssl_no_check_certificate else "true") + "\n")
88- if globals.ssl_cacert_file:
89- os.write(self.tempfile, "set ssl:ca-file '" + globals.ssl_cacert_file + "'\n")
90+ if self.cacert_file:
91+ os.write(self.tempfile, "set ssl:ca-file " + cmd_quote(self.cacert_file) + "\n")
92+ if globals.ssl_cacert_path:
93+ os.write(self.tempfile, "set ssl:ca-path " + cmd_quote(globals.ssl_cacert_path) + "\n")
94 if self.parsed_url.scheme == 'ftps':
95 os.write(self.tempfile, "set ftp:ssl-allow true\n")
96 os.write(self.tempfile, "set ftp:ssl-protect-data true\n")
97@@ -134,13 +131,14 @@
98 else:
99 os.write(self.tempfile, "open %s %s\n" % (self.authflag, self.url_string))
100 os.close(self.tempfile)
101+ # print settings in debug mode
102 if log.getverbosity() >= log.DEBUG:
103 f = open(self.tempname, 'r')
104 log.Debug("SETTINGS: \n"
105- "%s" % f.readlines())
106+ "%s" % f.read())
107
108 def _put(self, source_path, remote_filename):
109- commandline = "lftp -c 'source %s; mkdir -p %s; put %s -o %s'" % (
110+ commandline = "lftp -c \"source %s; mkdir -p %s; put %s -o %s\"" % (
111 self.tempname,
112 cmd_quote(self.remote_path),
113 cmd_quote(source_path.name),
114@@ -155,8 +153,8 @@
115 "%s" % (l))
116
117 def _get(self, remote_filename, local_path):
118- commandline = "lftp -c 'source %s; get %s -o %s'" % (
119- self.tempname,
120+ commandline = "lftp -c \"source %s; get %s -o %s\"" % (
121+ cmd_quote(self.tempname),
122 cmd_quote(self.remote_path) + remote_filename,
123 cmd_quote(local_path.name)
124 )
125@@ -172,9 +170,11 @@
126 # remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
127 remote_dir = urllib.unquote(self.parsed_url.path)
128 # print remote_dir
129- commandline = "lftp -c 'source %s; cd %s || exit 0; ls'" % (
130- self.tempname,
131- cmd_quote(self.remote_path)
132+ quoted_path = cmd_quote(self.remote_path);
133+ # failing to cd into the folder might be because it was not created already
134+ commandline = "lftp -c \"source %s; ( cd %s && ls ) || ( mkdir -p %s && cd %s && ls )\"" % (
135+ cmd_quote(self.tempname),
136+ quoted_path, quoted_path, quoted_path
137 )
138 log.Debug("CMD: %s" % commandline)
139 _, l, e = self.subprocess_popen(commandline)
140@@ -187,10 +187,10 @@
141 return [x.split()[-1] for x in l.split('\n') if x]
142
143 def _delete(self, filename):
144- commandline = "lftp -c 'source %s; cd %s; rm %s'" % (
145- self.tempname,
146+ commandline = "lftp -c \"source %s; cd %s; rm %s\"" % (
147+ cmd_quote(self.tempname),
148 cmd_quote(self.remote_path),
149- filename
150+ cmd_quote(filename)
151 )
152 log.Debug("CMD: %s" % commandline)
153 _, l, e = self.subprocess_popen(commandline)
154@@ -204,10 +204,10 @@
155 duplicity.backend.register_backend("fish", LFTPBackend)
156 duplicity.backend.register_backend("ftpes", LFTPBackend)
157
158-
159 duplicity.backend.register_backend("lftp+ftp", LFTPBackend)
160 duplicity.backend.register_backend("lftp+ftps", LFTPBackend)
161 duplicity.backend.register_backend("lftp+fish", LFTPBackend)
162+duplicity.backend.register_backend("lftp+ftpes", LFTPBackend)
163 duplicity.backend.register_backend("lftp+sftp", LFTPBackend)
164 duplicity.backend.register_backend("lftp+webdav", LFTPBackend)
165 duplicity.backend.register_backend("lftp+webdavs", LFTPBackend)
166@@ -216,7 +216,8 @@
167
168 duplicity.backend.uses_netloc.extend(['ftp', 'ftps', 'fish', 'ftpes',
169 'lftp+ftp', 'lftp+ftps',
170- 'lftp+fish', 'lftp+sftp',
171+ 'lftp+fish', 'lftp+ftpes',
172+ 'lftp+sftp',
173 'lftp+webdav', 'lftp+webdavs',
174 'lftp+http', 'lftp+https']
175 )
176
177=== modified file 'duplicity/backends/webdavbackend.py'
178--- duplicity/backends/webdavbackend.py 2016-02-05 09:58:57 +0000
179+++ duplicity/backends/webdavbackend.py 2016-03-01 16:25:36 +0000
180@@ -58,31 +58,25 @@
181 import socket
182 import ssl
183 except ImportError:
184- raise FatalBackendException("Missing socket or ssl libraries.")
185+ raise FatalBackendException(_("Missing socket or ssl python modules."))
186
187 httplib.HTTPSConnection.__init__(self, *args, **kwargs)
188
189 self.cacert_file = globals.ssl_cacert_file
190- cacert_candidates = ["~/.duplicity/cacert.pem",
191+ self.cacert_candidates = ["~/.duplicity/cacert.pem",
192 "~/duplicity_cacert.pem",
193 "/etc/duplicity/cacert.pem"]
194- #
195+ # if no cacert file was given search default locations
196 if not self.cacert_file:
197- for path in cacert_candidates:
198+ for path in self.cacert_candidates:
199 path = os.path.expanduser(path)
200 if (os.path.isfile(path)):
201 self.cacert_file = path
202 break
203- # still no cacert file, inform user
204- if not self.cacert_file:
205- raise FatalBackendException("""\
206-For certificate verification a cacert database file is needed in one of these locations: %s
207-Hints:
208- Consult the man page, chapter 'SSL Certificate Verification'.
209- Consider using the options --ssl-cacert-file, --ssl-no-check-certificate .""" % ", ".join(cacert_candidates))
210+
211 # check if file is accessible (libssl errors are not very detailed)
212- if not os.access(self.cacert_file, os.R_OK):
213- raise FatalBackendException("Cacert database file '%s' is not readable." % self.cacert_file)
214+ if self.cacert_file and not os.access(self.cacert_file, os.R_OK):
215+ raise FatalBackendException(_("Cacert database file '%s' is not readable.") % self.cacert_file)
216
217 def connect(self):
218 # create new socket
219@@ -92,8 +86,24 @@
220 self.sock = sock
221 self.tunnel()
222
223- # wrap the socket in ssl using verification
224- self.sock = ssl.wrap_socket(sock,
225+ # python 2.7.9+ supports default system certs now
226+ if "create_default_context" in dir(ssl):
227+ context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH, cafile=self.cacert_file, capath=globals.ssl_cacert_path)
228+ self.sock = context.wrap_socket(sock, server_hostname=self.host)
229+ # the legacy way needing a cert file
230+ else:
231+ if globals.ssl_cacert_path:
232+ raise FatalBackendException(_("Option '--ssl-cacert-path' is not supported with python 2.7.8 and below."))
233+
234+ if not self.cacert_file:
235+ raise FatalBackendException(_("""\
236+For certificate verification with python 2.7.8 or earlier a cacert database file is needed in one of these locations: %s
237+Hints:
238+ Consult the man page, chapter 'SSL Certificate Verification'.
239+ Consider using the options --ssl-cacert-file, --ssl-no-check-certificate .""") % ", ".join(self.cacert_candidates))
240+
241+ # wrap the socket in ssl using verification
242+ self.sock = ssl.wrap_socket(sock,
243 cert_reqs=ssl.CERT_REQUIRED,
244 ca_certs=self.cacert_file,
245 )
246@@ -129,9 +139,9 @@
247 self.password = self.get_password()
248 self.directory = self.sanitize_path(parsed_url.path)
249
250- log.Info("Using WebDAV protocol %s" % (globals.webdav_proto,))
251- log.Info("Using WebDAV host %s port %s" % (parsed_url.hostname, parsed_url.port))
252- log.Info("Using WebDAV directory %s" % (self.directory,))
253+ log.Info(_("Using WebDAV protocol %s") % (globals.webdav_proto,))
254+ log.Info(_("Using WebDAV host %s port %s") % (parsed_url.hostname, parsed_url.port))
255+ log.Info(_("Using WebDAV directory %s") % (self.directory,))
256
257 self.conn = None
258
259@@ -162,7 +172,7 @@
260 and self.conn.host == self.parsed_url.hostname:
261 return
262
263- log.Info("WebDAV create connection on '%s'" % (self.parsed_url.hostname))
264+ log.Info(_("WebDAV create connection on '%s'") % (self.parsed_url.hostname))
265 self._close()
266 # http schemes needed for redirect urls from servers
267 if self.parsed_url.scheme in ['webdav', 'http']:
268@@ -173,7 +183,7 @@
269 else:
270 self.conn = VerifiedHTTPSConnection(self.parsed_url.hostname, self.parsed_url.port)
271 else:
272- raise FatalBackendException("WebDAV Unknown URI scheme: %s" % (self.parsed_url.scheme))
273+ raise FatalBackendException(_("WebDAV Unknown URI scheme: %s") % (self.parsed_url.scheme))
274
275 def _close(self):
276 if self.conn:
277@@ -192,34 +202,34 @@
278 if self.digest_challenge is not None:
279 self.headers['Authorization'] = self.get_digest_authorization(path)
280
281- log.Info("WebDAV %s %s request with headers: %s " % (method, quoted_path, self.headers))
282- log.Info("WebDAV data length: %s " % len(str(data)))
283+ log.Info(_("WebDAV %s %s request with headers: %s ") % (method, quoted_path, self.headers))
284+ log.Info(_("WebDAV data length: %s ") % len(str(data)))
285 self.conn.request(method, quoted_path, data, self.headers)
286 response = self.conn.getresponse()
287- log.Info("WebDAV response status %s with reason '%s'." % (response.status, response.reason))
288+ log.Info(_("WebDAV response status %s with reason '%s'.") % (response.status, response.reason))
289 # resolve redirects and reset url on listing requests (they usually come before everything else)
290 if response.status in [301, 302] and method == 'PROPFIND':
291 redirect_url = response.getheader('location', None)
292 response.close()
293 if redirect_url:
294- log.Notice("WebDAV redirect to: %s " % urllib.unquote(redirect_url))
295+ log.Notice(_("WebDAV redirect to: %s ") % urllib.unquote(redirect_url))
296 if redirected > 10:
297- raise FatalBackendException("WebDAV redirected 10 times. Giving up.")
298+ raise FatalBackendException(_("WebDAV redirected 10 times. Giving up."))
299 self.parsed_url = duplicity.backend.ParsedUrl(redirect_url)
300 self.directory = self.sanitize_path(self.parsed_url.path)
301 return self.request(method, self.directory, data, redirected + 1)
302 else:
303- raise FatalBackendException("WebDAV missing location header in redirect response.")
304+ raise FatalBackendException(_("WebDAV missing location header in redirect response."))
305 elif response.status == 401:
306 response.read()
307 response.close()
308 self.headers['Authorization'] = self.get_authorization(response, quoted_path)
309- log.Info("WebDAV retry request with authentification headers.")
310- log.Info("WebDAV %s %s request2 with headers: %s " % (method, quoted_path, self.headers))
311- log.Info("WebDAV data length: %s " % len(str(data)))
312+ log.Info(_("WebDAV retry request with authentification headers."))
313+ log.Info(_("WebDAV %s %s request2 with headers: %s ") % (method, quoted_path, self.headers))
314+ log.Info(_("WebDAV data length: %s ") % len(str(data)))
315 self.conn.request(method, quoted_path, data, self.headers)
316 response = self.conn.getresponse()
317- log.Info("WebDAV response2 status %s with reason '%s'." % (response.status, response.reason))
318+ log.Info(_("WebDAV response2 status %s with reason '%s'.") % (response.status, response.reason))
319
320 return response
321
322@@ -337,11 +347,11 @@
323 log.Info("Checking existence dir %s: %d" % (d, response.status))
324
325 if response.status == 404:
326- log.Info("Creating missing directory %s" % d)
327+ log.Info(_("Creating missing directory %s") % d)
328
329 res = self.request("MKCOL", d)
330 if res.status != 201:
331- raise BackendException("WebDAV MKCOL %s failed: %s %s" % (d, res.status, res.reason))
332+ raise BackendException(_("WebDAV MKCOL %s failed: %s %s") % (d, res.status, res.reason))
333
334 def taste_href(self, href):
335 """
336@@ -353,8 +363,8 @@
337 raw_filename = self.getText(href.childNodes).strip()
338 parsed_url = urlparse.urlparse(urllib.unquote(raw_filename))
339 filename = parsed_url.path
340- log.Debug("webdav path decoding and translation: "
341- "%s -> %s" % (raw_filename, filename))
342+ log.Debug(_("WebDAV path decoding and translation: "
343+ "%s -> %s") % (raw_filename, filename))
344
345 # at least one WebDAV server returns files in the form
346 # of full URL:s. this may or may not be
347@@ -397,7 +407,7 @@
348 status = response.status
349 reason = response.reason
350 response.close()
351- raise BackendException("Bad status code %s reason %s." % (status, reason))
352+ raise BackendException(_("WebDAV GET Bad status code %s reason %s.") % (status, reason))
353 except Exception as e:
354 raise e
355 finally:
356@@ -418,7 +428,7 @@
357 status = response.status
358 reason = response.reason
359 response.close()
360- raise BackendException("Bad status code %s reason %s." % (status, reason))
361+ raise BackendException(_("WebDAV PUT Bad status code %s reason %s.") % (status, reason))
362 except Exception as e:
363 raise e
364 finally:
365@@ -437,7 +447,7 @@
366 status = response.status
367 reason = response.reason
368 response.close()
369- raise BackendException("Bad status code %s reason %s." % (status, reason))
370+ raise BackendException(_("WebDAV DEL Bad status code %s reason %s.") % (status, reason))
371 except Exception as e:
372 raise e
373 finally:
374
375=== modified file 'duplicity/commandline.py'
376--- duplicity/commandline.py 2016-02-10 17:15:49 +0000
377+++ duplicity/commandline.py 2016-03-01 16:25:36 +0000
378@@ -582,9 +582,9 @@
379 # user added ssh options
380 parser.add_option("--ssh-options", action="extend", metavar=_("options"))
381
382- # user added ssl options (webdav backend)
383+ # user added ssl options (used by webdav, lftp backend)
384 parser.add_option("--ssl-cacert-file", metavar=_("pem formatted bundle of certificate authorities"))
385-
386+ parser.add_option("--ssl-cacert-path", metavar=_("path to a folder with certificate authority files"))
387 parser.add_option("--ssl-no-check-certificate", action="store_true")
388
389 # Working directory for the tempfile module. Defaults to /tmp on most systems.
390
391=== modified file 'duplicity/globals.py'
392--- duplicity/globals.py 2015-10-10 00:02:35 +0000
393+++ duplicity/globals.py 2016-03-01 16:25:36 +0000
394@@ -242,8 +242,9 @@
395 # default cf backend is pyrax
396 cf_backend = "pyrax"
397
398-# HTTPS ssl optons (currently only webdav)
399+# HTTPS ssl options (currently only webdav, lftp)
400 ssl_cacert_file = None
401+ssl_cacert_path = None
402 ssl_no_check_certificate = False
403
404 # user added rsync options

Subscribers

People subscribed via source and target branches