Chromium Browser

Merge lp:~chromium-team/chromium-browser/chromium-translations-tools.head into lp:chromium-browser

  1. chromium-translations-tools.head
  2. Merge into chromium-browser.head
Proposed by Samuel Ramos
Status: Rejected
Rejected by: Nathan Teodosio
Proposed branch: lp:~chromium-team/chromium-browser/chromium-translations-tools.head
Merge into: lp:chromium-browser
Diff against target: 3433 lines (+3409/-0)
5 files modified
chromium2pot.py (+2610/-0)
create-patches.sh (+185/-0)
desktop2gettext.py (+378/-0)
update-inspector.py (+149/-0)
update-pot.sh (+87/-0)
To merge this branch: bzr merge lp:~chromium-team/chromium-browser/chromium-translations-tools.head
Reviewer Review Type Date Requested Status
Chromium team Pending
Review via email: mp+461443@code.launchpad.net
To post a comment you must log in.

Unmerged revisions

121. By Chad Miller

Handle GRD partial files.

Ignore "external" references, which are usually images.

120. By Ken VanDine

handled latest grd format

119. By Micah Gersten

* Temporarily workaround muliple bg locales in generated_resources

118. By Cris Dywan

* Add temporary workaround for new type not being used yet

117. By Fabien Tassin

* Add some helper scripts

116. By Fabien Tassin

* When updating common.gypi, fold 'locales' by size (instead of by groups of 10)

115. By Fabien Tassin

* Fix a regression introduced by the new fake-bidi pseudo locale
  (see https://sites.google.com/a/chromium.org/dev/Home/fake-bidi and
   http://code.google.com/p/chromium/issues/detail?id=73052)

114. By Fabien Tassin

* Add a --map-template-names knob allowing to handle renamed templates
  in some branches

113. By Fabien Tassin

* Add support for 'string-enum' and 'int-enum' outside of 'group' policies
  (needed since http://codereview.chromium.org/7287001/ landed)

112. By Fabien Tassin

* Move all new xtb files to third_party/launchpad_translations (relative to $SRC)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'chromium2pot.py'
2--- chromium2pot.py 1970-01-01 00:00:00 +0000
3+++ chromium2pot.py 2024-02-28 12:58:47 +0000
4@@ -0,0 +1,2610 @@
5+#!/usr/bin/python
6+# -*- coding: utf-8 -*-
7+
8+# (c) 2010-2011, Fabien Tassin <fta@ubuntu.com>
9+
10+# Convert grd/xtb files into pot/po for integration into the Launchpad
11+# translation system
12+
13+## grd files contain the strings for the 'pot' file(s).
14+## Keys are alphabetical (IDS_XXX).
15+# Sources:
16+# - $SRC/chrome/app/*.grd
17+# - $SRC/webkit/glue/*.grd
18+
19+## xtb files are referenced to by the grd files. They contain the translated
20+## strings for the 'po' our files. Keys are numerical (64bit ids).
21+# Sources:
22+# - $SRC/chrome/app/resources/*.xtb
23+# - $SRC/webkit/glue/resources/*.xtb
24+# and for launchpad contributed strings that already landed:
25+# - $SRC/third_party/launchpad_translations/*.xtb
26+
27+## the mapping between those keys is done using FingerPrint()
28+## [ taken from grit ] on a stripped version of the untranslated string
29+
30+## grd files contain a lot of <if expr="..."> (python-like) conditions.
31+## Evaluate those expressions but only skip strings with a lang restriction.
32+## For all other conditions (os, defines), simply expose them so translators
33+## know when a given string is expected.
34+
35+## TODO: handle <message translateable="false">
36+
37+import os, sys, shutil, re, getopt, codecs, urllib
38+from xml.dom import minidom
39+from xml.sax.saxutils import unescape
40+from datetime import datetime
41+from difflib import unified_diff
42+import textwrap, filecmp, json
43+
44+lang_mapping = {
45+ 'no': 'nb', # 'no' is obsolete and the more specific 'nb' (Norwegian Bokmal)
46+ # and 'nn' (Norwegian Nynorsk) are preferred.
47+ 'pt-PT': 'pt'
48+}
49+
50+####
51+# vanilla from $SRC/tools/grit/grit/extern/FP.py (r10982)
52+# See svn log http://src.chromium.org/svn/trunk/src/tools/grit/grit/extern/FP.py
53+
54+# Copyright (c) 2006-2008 The Chromium Authors. All rights reserved.
55+# Use of this source code is governed by a BSD-style license that can be
56+# found in the LICENSE file.
57+
58+try:
59+ import hashlib
60+ _new_md5 = hashlib.md5
61+except ImportError:
62+ import md5
63+ _new_md5 = md5.new
64+
65+def UnsignedFingerPrint(str, encoding='utf-8'):
66+ """Generate a 64-bit fingerprint by taking the first half of the md5
67+ of the string."""
68+ hex128 = _new_md5(str).hexdigest()
69+ int64 = long(hex128[:16], 16)
70+ return int64
71+
72+def FingerPrint(str, encoding='utf-8'):
73+ fp = UnsignedFingerPrint(str, encoding=encoding)
74+ # interpret fingerprint as signed longs
75+ if fp & 0x8000000000000000L:
76+ fp = - ((~fp & 0xFFFFFFFFFFFFFFFFL) + 1)
77+ return fp
78+####
79+
80+class EvalConditions:
81+ """ A class allowing an <if expr="xx"/> to be evaluated, based on an array of defines,
82+ a dict of local variables.
83+ As of Chromium 10:
84+ - the known defines are:
85+ [ 'chromeos', '_google_chrome', 'toolkit_views', 'touchui', 'use_titlecase' ]
86+ On Linux, only [ 'use_titlecase' ] is set.
87+ - the known variables are:
88+ 'os' ('linux2' on Linux)
89+ 'lang'
90+ See http://src.chromium.org/svn/trunk/src/build/common.gypi
91+ """
92+
93+ def eval(self, expression, defines = [ 'use_titlecase' ], vars = { 'os': "linux2" }):
94+
95+ def pp_ifdef(match):
96+ return str(match.group(1) in defines)
97+
98+ # evaluate all ifdefs
99+ expression = re.sub(r"pp_ifdef\('(.*?)'\)", pp_ifdef, expression)
100+ # evaluate the whole expression using the vars dict
101+ vars['__builtins__'] = { 'True': True, 'False': False } # prevent eval from using the real current globals
102+ return eval(expression, vars)
103+
104+ def lang_eval(self, expression, lang):
105+ """ only evaluate the expression against the lang, ignore all defines and other variables.
106+ This is needed to ignore a string that has lang restrictions (numerals, plurals, ..) but
107+ still keep it even if it's OS or defined don't match the local platform.
108+ """
109+ conditions = [ x for x in re.split(r'\s+(and|or)\s+', expression) if x.find('lang') >= 0 ]
110+ if len(conditions) == 0:
111+ return True
112+ assert len(conditions) == 1, "Expression '%s' has multiple lang conditions" % expression
113+ vars = { 'lang': lang, '__builtins__': { 'True': True, 'False': False } }
114+ return eval(conditions[0], vars)
115+
116+ def test(self):
117+ data = [
118+ { 'expr': "lang == 'ar'",
119+ 'vars': { 'lang': 'ar' },
120+ 'result': True
121+ },
122+ { 'expr': "lang == 'ar'",
123+ 'vars': { 'lang': 'fr' },
124+ 'result': False
125+ },
126+ { 'expr': "lang in ['ar', 'ro', 'lv']",
127+ 'vars': { 'lang': 'ar' },
128+ 'result': True
129+ },
130+ { 'expr': "lang in ['ar', 'ro', 'lv']",
131+ 'vars': { 'lang': 'pt-BR' },
132+ 'result': False
133+ },
134+ { 'expr': "lang not in ['ar', 'ro', 'lv']",
135+ 'vars': { 'lang': 'ar' },
136+ 'result': False
137+ },
138+ { 'expr': "lang not in ['ar', 'ro', 'lv']",
139+ 'vars': { 'lang': 'no' },
140+ 'result': True
141+ },
142+ { 'expr': "os != 'linux2' and os != 'darwin' and os.find('bsd') == -1",
143+ 'vars': { 'lang': 'no', 'os': 'bsdos' },
144+ 'result': False,
145+ 'lresult': True # no lang restriction in 'expr', so 'no' is ok
146+ },
147+ { 'expr': "os != 'linux2' and os != 'darwin' and os.find('bsd') > -1",
148+ 'vars': { 'lang': 'no', 'os': 'bsdos' },
149+ 'result': True,
150+ },
151+ { 'expr': "not pp_ifdef('chromeos')",
152+ 'vars': { 'lang': 'no' },
153+ 'defines': [],
154+ 'result': True,
155+ },
156+ { 'expr': "not pp_ifdef('chromeos')",
157+ 'vars': { 'lang': 'no' },
158+ 'defines': [ 'chromeos' ],
159+ 'result': False,
160+ 'lresult': True # no lang restriction in 'expr', so 'no' is ok
161+ },
162+ { 'expr': "pp_ifdef('_google_chrome') and (os == 'darwin')",
163+ 'vars': { 'lang': 'no', 'os': 'linux2' },
164+ 'defines': [ 'chromeos' ],
165+ 'result': False,
166+ 'lresult': True # no lang restriction in 'expr', so 'no' is ok
167+ },
168+ { 'expr': "pp_ifdef('_google_chrome') and (os == 'darwin')",
169+ 'vars': { 'lang': 'no', 'os': 'darwin' },
170+ 'defines': [ '_google_chrome' ],
171+ 'result': True
172+ },
173+ { 'expr': "not pp_ifdef('chromeos') and pp_ifdef('_google_chrome') and 'pt-PT' == lang",
174+ 'vars': { 'lang': 'pt-PT', 'os': 'darwin' },
175+ 'defines': [ '_google_chrome' ],
176+ 'result': True
177+ },
178+ { 'expr': "not pp_ifdef('chromeos') and pp_ifdef('_google_chrome') and 'pt-PT' == lang",
179+ 'vars': { 'lang': 'pt-PT', 'os': 'darwin' },
180+ 'defines': [ ],
181+ 'result': False,
182+ 'lresult': True
183+ },
184+ ]
185+ i = -1
186+ for d in data:
187+ i += 1
188+ defines = d['defines'] if 'defines' in d else []
189+ vars = d['vars'] if 'vars' in d else {}
190+ lvars = vars.copy() # make a copy because eval modifies it
191+ res = self.eval(d['expr'], defines = defines, vars = lvars)
192+ assert res == d['result'], "FAILED %d: expr: \"%s\" returned %s with vars = %s and defines = %s" % \
193+ (i, d['expr'], repr(res), repr(vars), repr(defines))
194+ print "All %d tests passed for EvalConditions.eval()" % (i + 1)
195+ i = -1
196+ for d in data:
197+ i += 1
198+ assert 'lang' in vars, "All test must have a 'lang' in 'vars', test %d doesn't: %s" % (i, repr(d))
199+ res = self.lang_eval(d['expr'], lang = d['vars']['lang'])
200+ expected = d['lresult'] if 'lresult' in d else d['result']
201+ assert res == expected, "FAILED %d: expr: \"%s\" returned %s with lang = %s for the lang_eval test" % \
202+ (i, d['expr'], repr(res), d['vars']['lang'])
203+ print "All %d tests passed for EvalConditions.lang_eval()" % (i + 1)
204+
205+class StringCvt:
206+ """ A class converting grit formatted strings to gettext back and forth.
207+ The idea is to always have:
208+ a/ grd2gettext(xtb2gettext(s)) == s
209+ b/ xtb2gettext(s) produces a string that the msgfmt checker likes and
210+ that makes sense to translators
211+ c/ grd2gettext(s) produces a string acceptable by upstream
212+ """
213+
214+ def xtb2gettext(self, string):
215+ """ parse the xtb (xml encoded) string and convert it to a gettext string """
216+
217+ def fold(string):
218+ return textwrap.wrap(string, break_long_words=False, width=76, drop_whitespace=False,
219+ expand_tabs=False, replace_whitespace=False, break_on_hyphens=False)
220+
221+ s = string.replace('\\n', '\\\\n')
222+ # escape all single '\' (not followed by 'n')
223+ s = re.sub(r'(?<!\\)(\\[^n\\\\])', r'\\\1', s)
224+ # remove all xml encodings
225+ s = self.unescape_xml(s)
226+ # replace '<ph name="FOO"/>' by '%{FOO}'
227+ s = re.sub(r'<ph name="(.*?)"/>', r'%{\1}', s)
228+ # fold
229+ # 1/ fold at \n
230+ # 2/ fold each part at ~76 char
231+ v = []
232+ ll = s.split('\n')
233+ sz = len(ll)
234+ if sz > 1:
235+ i = 0
236+ for l in ll:
237+ i += 1
238+ if i == sz:
239+ v.extend(fold(l))
240+ else:
241+ v.extend(fold(l + '\\n'))
242+ else:
243+ v.extend(fold(ll[0]))
244+ if len(v) > 1:
245+ v[:0] = [ '' ]
246+ s = '"' + '"\n"'.join(v) + '"'
247+ return s
248+
249+ def decode_xml_entities(self, string):
250+ def replace_xmlent(match):
251+ if match.group(1)[:1] == 'x':
252+ return unichr(int("0" + match.group(1), 16))
253+ else:
254+ return unichr(int(match.group(1)))
255+
256+ return re.sub(r'&#(x\w+|\d+);', replace_xmlent, string)
257+
258+ def unescape_xml(self, string):
259+ string = unescape(string).replace('&quot;', '\\"').replace('&apos;', "'")
260+ string = self.decode_xml_entities(string)
261+ return string
262+
263+ def grd2gettext(self, string):
264+ """ parse the string returned from minidom and convert it to a gettext string.
265+ This is similar to str_cvt_xtb2gettext but minidom has its own magic for encoding
266+ """
267+ return self.xtb2gettext(string)
268+
269+ def gettext2xtb(self, string):
270+ """ parse the gettext string and convert it to an xtb (xml encoded) string. """
271+ u = []
272+ for s in string.split(u'\n'):
273+ # remove the enclosing double quotes
274+ u.append(s[1:][:-1])
275+ s = u"".join(u)
276+
277+ # encode the xml special chars
278+ s = s.replace("&", "&amp;") # must be first!
279+ s = s.replace("<", "&lt;")
280+ s = s.replace(">", "&gt;")
281+ s = s.replace('\\"', "&quot;")
282+ # special case, html comments
283+ s = re.sub(r'&lt;!--(.*?)--&gt;', r'<!--\1-->', s, re.S)
284+ # replace non-ascii by &#xxx; codes
285+ # s = s.encode("ascii", "xmlcharrefreplace")
286+ # replace '%{FOO}' by '<ph name="FOO"/>'
287+ s = re.sub(r'%{(.*?)}', r'<ph name="\1"/>', s)
288+ # unquote \\n and \\\\n
289+ s = re.sub(r'(?<!\\)\\n', r'\n', s)
290+ # unquote all control chars
291+ s = re.sub(r'\\\\([^\\])', r'\\\1', s)
292+
293+ # launchpad seems to always quote tabs
294+ s = s.replace("\\t", "\t")
295+ return s
296+
297+ def test(self):
298+ # unit tests
299+ data = [
300+ # tab
301+ { 'id': '0',
302+ 'xtb': u'foo bar',
303+ 'po': u'"foo bar"' },
304+ { 'id': '1',
305+ 'xtb': u'foo\tbar',
306+ 'po': u'"foo\tbar"' },
307+ # &amp;
308+ { 'id': '6779164083355903755',
309+ 'xtb': u'Supprime&amp;r',
310+ 'po': u'"Supprime&r"' },
311+ # &quot;
312+ { 'id': '4194570336751258953',
313+ 'xtb': u'Activer la fonction &quot;taper pour cliquer&quot;',
314+ 'po': u'"Activer la fonction \\"taper pour cliquer\\""' },
315+ # &lt; / &gt;
316+ { 'id': '7615851733760445951',
317+ 'xtb': u'&lt;aucun cookie sélectionné&gt;',
318+ 'po': u'"<aucun cookie sélectionné>"' },
319+ # <ph name="FOO"/>
320+ { 'id': '5070288309321689174',
321+ 'xtb': u'<ph name="EXTENSION_NAME"/> :',
322+ 'po': u'"%{EXTENSION_NAME} :"' },
323+ { 'id': '1467071896935429871',
324+ 'xtb': u'Téléchargement de la mise à jour du système : <ph name="PERCENT"/>% terminé',
325+ 'po': u'"Téléchargement de la mise à jour du système : %{PERCENT}% terminé"' },
326+ # line folding
327+ { 'id': '1526811905352917883',
328+ 'xtb': u'Une nouvelle tentative de connexion avec SSL 3.0 a dû être effectuée. Cette opération indique généralement que le serveur utilise un logiciel très ancien et qu\'il est susceptible de présenter d\'autres problèmes de sécurité.',
329+ 'po': u'""\n"Une nouvelle tentative de connexion avec SSL 3.0 a dû être effectuée. Cette "\n"opération indique généralement que le serveur utilise un logiciel très "\n"ancien et qu\'il est susceptible de présenter d\'autres problèmes de sécurité."' },
330+ { 'id': '7999229196265990314',
331+ 'xtb': u'Les fichiers suivants ont été créés :\n\nExtension : <ph name="EXTENSION_FILE"/>\nFichier de clé : <ph name="KEY_FILE"/>\n\nConservez votre fichier de clé en lieu sûr. Vous en aurez besoin lors de la création de nouvelles versions de l\'extension.',
332+ 'po': u'""\n"Les fichiers suivants ont été créés :\\n"\n"\\n"\n"Extension : %{EXTENSION_FILE}\\n"\n"Fichier de clé : %{KEY_FILE}\\n"\n"\\n"\n"Conservez votre fichier de clé en lieu sûr. Vous en aurez besoin lors de la "\n"création de nouvelles versions de l\'extension."' },
333+ # quoted LF
334+ { 'id': '4845656988780854088',
335+ 'xtb': u'Synchroniser uniquement les paramètres et\\ndonnées qui ont changé depuis la dernière connexion\\n(requiert votre mot de passe précédent)',
336+ 'po': u'""\n"Synchroniser uniquement les paramètres et\\\\ndonnées qui ont changé depuis la"\n" dernière connexion\\\\n(requiert votre mot de passe précédent)"' },
337+ { 'id': '1761265592227862828', # lang: 'el'
338+ 'xtb': u'Συγχρονισμός όλων των ρυθμίσεων και των δεδομένων\\n (ενδέχεται να διαρκέσει ορισμένο χρονικό διάστημα)',
339+ 'po': u'""\n"Συγχρονισμός όλων των ρυθμίσεων και των δεδομένων\\\\n (ενδέχεται να διαρκέσει"\n" ορισμένο χρονικό διάστημα)"' },
340+ { 'id': '1768211415369530011', # lang: 'de'
341+ 'xtb': u'Folgende Anwendung wird gestartet, wenn Sie diese Anforderung akzeptieren:\\n\\n <ph name="APPLICATION"/>',
342+ 'po': u'""\n"Folgende Anwendung wird gestartet, wenn Sie diese Anforderung "\n"akzeptieren:\\\\n\\\\n %{APPLICATION}"' },
343+ # weird controls
344+ { 'id': '5107325588313356747', # lang: 'es-419'
345+ 'xtb': u'Para ocultar el acceso a este programa, debes desinstalarlo. Para ello, utiliza\\n<ph name="CONTROL_PANEL_APPLET_NAME"/> del Panel de control.\\n\¿Deseas iniciar <ph name="CONTROL_PANEL_APPLET_NAME"/>?',
346+ 'po': u'""\n"Para ocultar el acceso a este programa, debes desinstalarlo. Para ello, "\n"utiliza\\\\n%{CONTROL_PANEL_APPLET_NAME} del Panel de control.\\\\n\\\\¿Deseas "\n"iniciar %{CONTROL_PANEL_APPLET_NAME}?"' }
347+ ]
348+
349+ for string in data:
350+ s = u"<x>" + string['xtb'] + u"</x>"
351+ s = s.encode('ascii', 'xmlcharrefreplace')
352+ dom = minidom.parseString(s)
353+ s = dom.firstChild.toxml()[3:][:-4]
354+ e = self.grd2gettext(s)
355+ if e != string['po']:
356+ assert False, "grd2gettext() failed for id " + string['id'] + \
357+ ". \nExpected: " + repr(string['po']) + "\nGot: " + repr(e)
358+ e = self.xtb2gettext(string['xtb'])
359+ if e != string['po']:
360+ assert False, "xtb2gettext() failed for id " + string['id'] + \
361+ ". \nExpected: " + repr(string['po']) + "\nGot: " + repr(e)
362+ u = self.gettext2xtb(e)
363+ if u != string['xtb']:
364+ assert False, "gettext2xtb() failed for id " + string['id'] + \
365+ ". \nExpected: " + repr(string['xtb']) + "\nGot: " + repr(u)
366+ print string['id'] + " ok"
367+
368+ # more tests with only po to xtb to test some weird launchpad po exports
369+ data2 = [
370+ { 'id': '1768211415369530011', # lang: 'de'
371+ 'po': u'""\n"Folgende Anwendung wird gestartet, wenn Sie diese Anforderung akzeptieren:\\\\"\n"n\\\\n %{APPLICATION}"',
372+ 'xtb': u'Folgende Anwendung wird gestartet, wenn Sie diese Anforderung akzeptieren:\\n\\n <ph name="APPLICATION"/>' },
373+ ]
374+ for string in data2:
375+ u = self.gettext2xtb(string['po'])
376+ if u != string['xtb']:
377+ assert False, "gettext2xtb() failed for id " + string['id'] + \
378+ ". \nExpected: " + repr(string['xtb']) + "\nGot: " + repr(u)
379+ print string['id'] + " ok"
380+
381+######
382+
383+class PotFile(dict):
384+ """
385+ Read and write gettext pot files
386+ """
387+
388+ def __init__(self, filename, date = None, debug = False, branch_name = "default", branch_dir = os.getcwd()):
389+ self.debug = debug
390+ self.lang = None
391+ self.filename = filename
392+ self.tfile = filename + ".new"
393+ self.branch_dir = branch_dir
394+ self.branch_name = branch_name
395+ self.template_date = date
396+ self.translation_date = "YEAR-MO-DA HO:MI+ZONE"
397+ self.is_pot = True
398+ self.fd = None
399+ self.fd_mode = "rb"
400+ if self.template_date is None:
401+ self.template_date = datetime.utcnow().strftime("%Y-%m-%d %H:%M+0000")
402+ self.strings = []
403+
404+ def add_string(self, id, comment, string, translation = "", origin = None):
405+ self.strings.append({ 'id': id, 'comment': comment, 'string': string,
406+ 'origin': origin, 'translation': translation })
407+
408+ def replace_file_if_newer(self):
409+ filename = os.path.join(self.branch_dir, self.filename) if self.branch_dir is not None \
410+ else self.filename
411+ tfile = os.path.join(self.branch_dir, self.tfile) if self.branch_dir is not None \
412+ else self.tfile
413+ if os.path.isfile(filename) and filecmp.cmp(filename, tfile) == 1:
414+ os.unlink(tfile)
415+ return 0
416+ else:
417+ os.rename(tfile, filename)
418+ return 1
419+
420+ def get_mtime(self, file):
421+ rfile = os.path.join(self.branch_dir, file)
422+ if self.debug:
423+ print "getmtime(%s) [%s]" % (file, os.path.abspath(rfile))
424+ return os.path.getmtime(rfile)
425+
426+ def open(self, mode = "rb", filename = None):
427+ if filename is not None:
428+ self.filename = filename
429+ self.tfile = filename + ".new"
430+ rfile = os.path.join(self.branch_dir, self.filename)
431+ rtfile = os.path.join(self.branch_dir, self.tfile)
432+ if self.fd is not None:
433+ self.close()
434+ self.fd_mode = mode
435+ if mode.find("r") != -1:
436+ if self.debug:
437+ print "open %s [mode=%s] from branch '%s' [%s]" % (self.filename, mode, self.branch_name, os.path.abspath(rfile))
438+ self.fd = codecs.open(rfile, mode, encoding="utf-8")
439+ else:
440+ if self.debug:
441+ print "open %s [mode=%s] from branch '%s' [%s]" % (self.tfile, mode, self.branch_name, os.path.abspath(rtfile))
442+ self.fd = codecs.open(rtfile, mode, encoding="utf-8")
443+
444+ def close(self):
445+ self.fd.close()
446+ self.fd = None
447+ if self.fd_mode.find("w") != -1:
448+ return self.replace_file_if_newer()
449+
450+ def read_string(self):
451+ string = {}
452+ cur = None
453+ while 1:
454+ s = self.fd.readline()
455+ if len(s) == 0 or s == "\n":
456+ break # EOF or end of block
457+ if s.rfind('\n') == len(s) - 1:
458+ s = s[:-1] # chomp
459+ if s.find("# ") == 0 or s == "#": # translator-comment
460+ if 'comment' not in string:
461+ string['comment'] = ''
462+ string['comment'] += s[2:]
463+ continue
464+ if s.find("#:") == 0: # reference
465+ if 'reference' not in string:
466+ string['reference'] = ''
467+ string['reference'] += s[2:]
468+ if s[2:].find(" id: ") == 0:
469+ string['id'] = s[7:].split(' ')[0]
470+ continue
471+ if s.find("#.") == 0: # extracted-comments
472+ if 'extracted' not in string:
473+ string['extracted'] = ''
474+ string['extracted'] += s[2:]
475+ if s[2:].find(" - condition: ") == 0:
476+ if 'conditions' not in string:
477+ string['conditions'] = []
478+ string['conditions'].append(s[16:])
479+ continue
480+ if s.find("#~") == 0: # obsolete messages
481+ continue
482+ if s.find("#") == 0: # something else
483+ print "%s not expected. Skip" % repr(s)
484+ continue # not supported/expected
485+ if s.find("msgid ") == 0:
486+ cur = "string"
487+ if cur not in string:
488+ string[cur] = u""
489+ else:
490+ string[cur] += "\n"
491+ string[cur] += s[6:]
492+ continue
493+ if s.find("msgstr ") == 0:
494+ cur = "translation"
495+ if cur not in string:
496+ string[cur] = u""
497+ else:
498+ string[cur] += "\n"
499+ string[cur] += s[7:]
500+ continue
501+ if s.find('"') == 0:
502+ if cur is None:
503+ print "'%s' not expected here. Skip" % s
504+ continue
505+ string[cur] += "\n" + s
506+ continue
507+ print "'%s' not expected here. Skip" % s
508+ return None if string == {} else string
509+
510+ def write(self, string):
511+ self.fd.write(string)
512+
513+ def write_header(self):
514+ lang_team = "LANGUAGE <LL@li.org>" if self.is_pot else "%s <%s@li.org>" % (self.lang, self.lang)
515+ lang_str = "template" if self.is_pot else "for lang '%s'" % self.lang
516+ date = "YEAR-MO-DA HO:MI+ZONE" if self.is_pot else \
517+ datetime.fromtimestamp(self.translation_date).strftime("%Y-%m-%d %H:%M+0000")
518+ self.write("# Chromium Translations %s.\n"
519+ "# Copyright (C) 2010-2011 Fabien Tassin\n"
520+ "# This file is distributed under the same license as the chromium-browser package.\n"
521+ "# Fabien Tassin <fta@ubuntu.com>, 2010-2011.\n"
522+ "#\n" % lang_str)
523+ # FIXME: collect contributors (can LP export them?)
524+ self.write('msgid ""\n'
525+ 'msgstr ""\n'
526+ '"Project-Id-Version: chromium-browser.head\\n"\n'
527+ '"Report-Msgid-Bugs-To: https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+filebug\\n"\n'
528+ '"POT-Creation-Date: %s\\n"\n'
529+ '"PO-Revision-Date: %s\\n"\n'
530+ '"Last-Translator: FULL NAME <EMAIL@ADDRESS>\\n"\n'
531+ '"Language-Team: %s\\n"\n'
532+ '"MIME-Version: 1.0\\n"\n'
533+ '"Content-Type: text/plain; charset=UTF-8\\n"\n'
534+ '"Content-Transfer-Encoding: 8bit\\n"\n\n' % \
535+ (datetime.fromtimestamp(self.template_date).strftime("%Y-%m-%d %H:%M+0000"),
536+ date, lang_team))
537+
538+ def write_footer(self):
539+ pass
540+
541+ def write_all_strings(self):
542+ for string in self.strings:
543+ self.write(u"#. %s\n" % u"\n#. ".join(string['comment'].split("\n")))
544+ self.write(u"#: id: %s (used in the following branches: %s)\n" % \
545+ (string['id'], ", ".join(string['origin'])))
546+ self.write(u'msgid %s\n' % StringCvt().xtb2gettext(string['string']))
547+ self.write(u'msgstr %s\n\n' % StringCvt().xtb2gettext(string['translation']))
548+
549+ def export_file(self, directory = None, filename = None):
550+ self.open(mode = "wb", filename = filename)
551+ self.write_header()
552+ self.write_all_strings()
553+ self.write_footer()
554+ return self.close()
555+
556+ def import_file(self):
557+ self.mtime = self.get_mtime(self.filename)
558+ self.open()
559+ while 1:
560+ string = self.read_string()
561+ if string is None:
562+ break
563+ self.strings.append(string)
564+ self.close()
565+
566+ def pack_comment(self, data):
567+ comment = ""
568+ for ent in sorted(data, lambda x,y: cmp(x['code'], y['code'])):
569+ comment += "%s\n- description: %s\n" % (ent['code'], ent['desc'])
570+ if ent['test'] is not None:
571+ comment += "- condition: %s\n" % ent['test']
572+ comment = comment[:-1] # strip trailing \n
573+ return comment
574+
575+ def get_origins(self, data):
576+ o = []
577+ for ent in sorted(data, lambda x,y: cmp(x['code'], y['code'])):
578+ for origin in ent['origin']:
579+ if origin not in o:
580+ o.append(origin)
581+ return o
582+
583+ def import_grd(self, grd):
584+ imported = 0
585+ for id in sorted(grd.supported_ids.keys()):
586+ if 'ids' not in grd.supported_ids[id]:
587+ continue
588+ comment = self.pack_comment(grd.supported_ids[id]['ids'])
589+ string = grd.supported_ids[id]['ids'][0]['val']
590+ origin = self.get_origins(grd.supported_ids[id]['ids'])
591+ self.strings.append({ 'id': id, 'comment': comment, 'string': string,
592+ 'origin': origin, 'translation': '' })
593+ imported += 1
594+ if self.debug:
595+ print "imported %d strings from the grd template" % imported
596+
597+class PoFile(PotFile):
598+ """
599+ Read and write gettext po files
600+ """
601+
602+ def __init__(self, lang, filename, template, date = None, debug = None,
603+ branch_name = "default", branch_dir = os.getcwd()):
604+ super(PoFile, self).__init__(filename, date = template.template_date, debug = debug,
605+ branch_name = branch_name, branch_dir = branch_dir)
606+ self.template = template
607+ self.lang = lang
608+ self.translation_date = date
609+ self.is_pot = False
610+
611+ def import_xtb(self, xtb):
612+ # only import strings present in the current template
613+ imported = 0
614+ for id in sorted(xtb.template.supported_ids.keys()):
615+ if 'ids' not in xtb.template.supported_ids[id]:
616+ continue
617+ translation = xtb.strings[id] if id in xtb.strings else ""
618+ comment = self.template.pack_comment(xtb.template.supported_ids[id]['ids'])
619+ string = xtb.template.supported_ids[id]['ids'][0]['val']
620+ origin = self.get_origins(xtb.template.supported_ids[id]['ids'])
621+ self.add_string(id, comment, string, translation, origin)
622+ imported += 1
623+ if self.debug:
624+ print "imported %d translations for lang %s from xtb into po %s" % (imported, self.lang, self.filename)
625+
626+class GrdFile(PotFile):
627+ """
628+ Read a Grit GRD file (write is not supported)
629+ """
630+ def __init__(self, filename, date = None, lang_mapping = None, debug = None,
631+ branch_name = "default", branch_dir = os.getcwd()):
632+ super(GrdFile, self).__init__(filename, date = date, debug = debug,
633+ branch_name = branch_name, branch_dir = branch_dir)
634+ self.lang_mapping = lang_mapping
635+ self.mapped_langs = {}
636+ self.supported_langs = {}
637+ self.supported_ids = {}
638+ self.supported_ids_counts = {}
639+ self.translated_strings = {}
640+ self.stats = {} # per lang
641+ self.debug = debug
642+ self._PH_REGEXP = re.compile('(<ph name=")([^"]*)("/>)')
643+
644+ def open(self):
645+ pass
646+
647+ def close(self):
648+ pass
649+
650+ def write_header(self):
651+ raise Exception("Not implemented!")
652+
653+ def write_footer(self):
654+ raise Exception("Not implemented!")
655+
656+ def write_all_strings(self):
657+ raise Exception("Not implemented!")
658+
659+ def export_file(self, directory = None, filename = None, global_langs = None, langs = None):
660+ fdi = codecs.open(self.filename, 'rb', encoding="utf-8")
661+ fdo = codecs.open(filename, 'wb', encoding="utf-8")
662+ # can't use minidom here as the file is manually generated and the
663+ # output will create big diffs. parse the source file line by line
664+ # and insert our xtb in the <translations> section. Also insert new
665+ # langs in the <outputs> section (with type="data_package" or type="js_map_format").
666+ # Let everything else untouched
667+ tr_found = False
668+ tr_saved = []
669+ tr_has_ifs = False
670+ tr_skipping_if_not = False
671+ pak_found = False
672+ pak_saved = []
673+ # langs, sorted by their xtb names
674+ our_langs = map(lambda x: x[0],
675+ sorted(map(lambda x: (x, self.mapped_langs[x]['xtb_file']),
676+ self.mapped_langs),
677+ key = lambda x: x[1])) # d'oh!
678+ if langs is None:
679+ langs = our_langs[:]
680+ for line in fdi.readlines():
681+ if re.match(r'.*?<output filename=".*?" type="(data_package|js_map_format)"', line):
682+ pak_found = True
683+ pak_saved.append(line)
684+ continue
685+ if line.find('</outputs>') > 0:
686+ pak_found = False
687+ ours = global_langs[:]
688+ chunks = {}
689+ c = None
690+ pak_if = None
691+ pak_is_in_if = False
692+ for l in pak_saved:
693+ if l.find("<!-- ") > 0:
694+ c = l
695+ continue
696+ if l.find("<if ") > -1:
697+ c = l if c is None else c + l
698+ tr_has_ifs = True
699+ pak_is_in_if = True
700+ continue
701+ if l.find("</if>") > -1:
702+ c = l if c is None else c + l
703+ pak_is_in_if = False
704+ continue
705+ m = re.match(r'.*?<output filename="(.*?)_([^_\.]+)\.(pak|js)" type="(data_package|js_map_format)" lang="(.*?)" />', l)
706+ if m is not None:
707+ x = { 'name': m.group(1), 'ext': m.group(3), 'lang': m.group(5), 'file_lang': m.group(2),
708+ 'type': m.group(4), 'in_if': pak_is_in_if, 'line': l }
709+ if c is not None:
710+ x['comment'] = c
711+ c = None
712+ k = m.group(2) if m.group(2) != 'nb' else 'no'
713+ chunks[k] = x
714+ else:
715+ if c is None:
716+ c = l
717+ else:
718+ c += l
719+ is_in_if = False
720+ for lang in sorted(chunks.keys()):
721+ tlang = lang if lang != 'no' else 'nb'
722+ while len(ours) > 0 and ((ours[0] == 'nb' and 'no' < tlang) or (ours[0] != 'nb' and ours[0] < tlang)):
723+ if ours[0] in chunks:
724+ ours = ours[1:]
725+ continue
726+ if tr_has_ifs and is_in_if is False:
727+ fdo.write(' <if expr="pp_ifdef(\'use_third_party_translations\')">\n')
728+ f = "%s_%s.%s" % (chunks[lang]['name'], ours[0], chunks[lang]['ext'])
729+ fdo.write(' %s<output filename="%s" type="%s" lang="%s" />\n' % \
730+ (' ' if tr_has_ifs else '', f, chunks[lang]['type'], ours[0]))
731+ is_in_if = True
732+ if tr_has_ifs and chunks[lang]['in_if'] is False:
733+ if 'comment' not in chunks[lang] or chunks[lang]['comment'].find('</if>') == -1:
734+ fdo.write(' </if>\n')
735+ is_in_if = False
736+ ours = ours[1:]
737+ if 'comment' in chunks[lang]:
738+ for s in chunks[lang]['comment'].split('\n')[:-1]:
739+ if chunks[lang]['in_if'] is True and is_in_if and s.find('<if ') > -1:
740+ continue
741+ if s.find('<!-- No translations available. -->') > -1:
742+ continue
743+ fdo.write(s + '\n')
744+ fdo.write(chunks[lang]['line'])
745+ if ours[0] == tlang:
746+ ours = ours[1:]
747+ is_in_if = chunks[lang]['in_if']
748+ if len(chunks.keys()) > 0:
749+ while len(ours) > 0:
750+ f = "%s_%s.%s" % (chunks[lang]['name'], ours[0], chunks[lang]['ext'])
751+ if tr_has_ifs and is_in_if is False:
752+ fdo.write(' <if expr="pp_ifdef(\'use_third_party_translations\')">\n')
753+ fdo.write(' %s<output filename="%s" type="data_package" lang="%s" />\n' % \
754+ (' ' if tr_has_ifs else '', f, ours[0]))
755+ is_in_if = True
756+ ours = ours[1:]
757+ if tr_has_ifs and is_in_if:
758+ fdo.write(' </if>\n')
759+ is_in_if = False
760+ if c is not None:
761+ for s in c.split('\n')[:-1]:
762+ if s.find('<!-- No translations available. -->') > -1:
763+ continue
764+ if s.find('</if>') > -1:
765+ continue
766+ fdo.write(s + '\n')
767+ if line.find('<translations>') > 0:
768+ fdo.write(line)
769+ tr_found = True
770+ continue
771+ if line.find('</translations>') > 0:
772+ tr_found = False
773+ ours = our_langs[:]
774+ chunks = {}
775+ obsolete = []
776+ c = None
777+ tr_if = None
778+ tr_is_in_if = False
779+ for l in tr_saved:
780+ if l.find("</if>") > -1:
781+ if tr_skipping_if_not:
782+ tr_skipping_if_not = False
783+ continue
784+ tr_is_in_if = False
785+ continue
786+ if tr_skipping_if_not:
787+ continue
788+ if l.find("<!-- ") > 0:
789+ c = l if c is None else c + l
790+ continue
791+ if l.find("<if ") > -1:
792+ m = re.match(r'.*?<if expr="not pp_ifdef\(\'use_third_party_translations\'\)"', l)
793+ if m is not None:
794+ tr_skipping_if_not = True
795+ continue
796+ tr_has_ifs = True
797+ tr_is_in_if = True
798+ continue
799+ m = re.match(r'.*?<file path=".*_([^_]+)\.xtb" lang="(.*?)"', l)
800+ if m is not None:
801+ tlang = m.group(2)
802+ if m.group(1) == 'iw':
803+ tlang = m.group(1)
804+ x = { 'lang': tlang, 'line': l, 'in_if': tr_is_in_if }
805+ if c is not None:
806+ x['comment'] = c
807+ c = None
808+ chunks[tlang] = x
809+ if tlang not in langs and tlang not in map(lambda t: self.mapped_langs[t]['grit'], langs):
810+ obsolete.append(tlang)
811+ else:
812+ if c is None:
813+ c = l
814+ else:
815+ c += l
816+ is_in_if = False
817+ # Do we want <if/> in the <translations/> block? (they are only mandatory in the <outputs/> block)
818+ want_ifs_in_translations = False
819+ for lang in sorted(chunks.keys()):
820+ while len(ours) > 0 and self.mapped_langs[ours[0]]['xtb_file'] < lang.replace('@', '-'):
821+ if ours[0] not in self.supported_langs:
822+ if self.debug:
823+ print "Skipped export of lang '%s' (most probably a 'po' file without any translated strings)" % ours[0]
824+ ours = ours[1:]
825+ continue
826+ if ours[0] in obsolete:
827+ if self.debug:
828+ print "Skipped export of lang '%s' (now obsolete)" % ours[0]
829+ ours = ours[1:]
830+ continue
831+ f = os.path.relpath(self.supported_langs[ours[0]], os.path.dirname(self.filename))
832+ if want_ifs_in_translations and tr_has_ifs and is_in_if is False:
833+ fdo.write(' <if expr="pp_ifdef(\'use_third_party_translations\')">\n')
834+ is_in_if = True
835+ fdo.write(' %s<file path="%s" lang="%s" />\n' %
836+ (' ' if (is_in_if or want_ifs_in_translations) and tr_has_ifs else '', f, ours[0]))
837+ if tr_has_ifs and chunks[lang]['in_if'] is False:
838+ if want_ifs_in_translations:
839+ fdo.write(' </if>\n')
840+ is_in_if = False
841+ ours = ours[1:]
842+ if 'comment' in chunks[lang]:
843+ for s in chunks[lang]['comment'].split('\n')[:-1]:
844+ if chunks[lang]['in_if'] is True and is_in_if and s.find('<if ') > -1:
845+ continue
846+ if s.find('<!-- No translations available. -->') > -1:
847+ continue
848+ fdo.write(s + '\n')
849+ if lang not in obsolete:
850+ fdo.write(chunks[lang]['line'])
851+ ours = ours[1:]
852+ is_in_if = chunks[lang]['in_if']
853+ while len(ours) > 0:
854+ if ours[0] in self.supported_langs:
855+ f = os.path.relpath(self.supported_langs[ours[0]], os.path.dirname(self.filename))
856+ if want_ifs_in_translations and tr_has_ifs and is_in_if is False:
857+ fdo.write(' <if expr="pp_ifdef(\'use_third_party_translations\')">\n')
858+ is_in_if = True
859+ fdo.write(' %s<file path="%s" lang="%s" />\n' %
860+ (' ' if (is_in_if or want_ifs_in_translations) and tr_has_ifs else '', f, ours[0]))
861+ elif self.debug:
862+ print "Skipped lang %s with no translated strings" % ours[0]
863+ ours = ours[1:]
864+
865+ if is_in_if and want_ifs_in_translations:
866+ fdo.write(' </if>\n')
867+ is_in_if = False
868+ if c is not None:
869+ for s in c.split('\n')[:-1]:
870+ if s.find('<!-- No translations available. -->') > -1:
871+ continue
872+ if s.find('</if>') > -1:
873+ continue
874+ fdo.write(s + '\n')
875+ if tr_found:
876+ tr_saved.append(line)
877+ continue
878+ if pak_found:
879+ pak_saved.append(line)
880+ continue
881+ fdo.write(line)
882+ fdi.close()
883+ fdo.close()
884+
885+ def uc(self, match):
886+ return match.group(2).upper()
887+
888+ def uc_name(self, match):
889+ return match.group(1) + match.group(2).upper() + match.group(3)
890+
891+ def is_string_valid_for_lang(self, id, lang):
892+ ok = False
893+ for string in self.supported_ids[id]['ids']:
894+ if string['test'] is not None:
895+ ok |= EvalConditions().lang_eval(string['test'], lang)
896+ if ok:
897+ break
898+ else:
899+ ok = True
900+ break
901+ return ok
902+
903+ def get_supported_strings_count(self, lang):
904+ # need to ignore strings for which this lang is not wanted in the <if> conditions
905+ if lang in self.supported_ids_counts:
906+ return self.supported_ids_counts[lang]['count'], self.supported_ids_counts[lang]['skipped']
907+ count = 0
908+ skipped = 0
909+ for id in self.supported_ids:
910+ ok = self.is_string_valid_for_lang(id, lang)
911+ if ok:
912+ count += 1
913+ else:
914+ skipped += 1
915+ assert count + skipped == len(self.supported_ids.keys())
916+ self.supported_ids_counts[lang] = { 'count': count, 'skipped': skipped }
917+ return count, skipped
918+
919+ def get_supported_langs(self):
920+ return sorted(self.supported_langs.keys())
921+
922+ def get_supported_lang_filenames(self):
923+ """ return the list of (xtb) filenames sorted by langs (so it's
924+ possible to zip() it) """
925+ return map(lambda l: self.supported_langs[l], sorted(self.supported_langs.keys()))
926+
927+ def update_stats(self, lang, translated_upstream = 0, obsolete = 0,
928+ new = 0, updated = 0, skipped_lang = 0, mandatory_linux = 0):
929+ if lang not in self.stats:
930+ self.stats[lang] = { 'translated_upstream': 0, 'skipped_lang': 0,
931+ 'obsolete': 0, 'new': 0, 'updated': 0,
932+ 'mandatory_linux': 0 }
933+ self.stats[lang]['translated_upstream'] += translated_upstream - updated
934+ self.stats[lang]['obsolete'] += obsolete
935+ self.stats[lang]['new'] += new
936+ self.stats[lang]['updated'] += updated
937+ self.stats[lang]['skipped_lang'] += skipped_lang
938+ self.stats[lang]['mandatory_linux'] += mandatory_linux
939+
940+ def merge_template(self, template, newer_preferred = True):
941+ """ merge strings from 'template' into self (the master template).
942+ If the string differs, prefer the new one when newer_preferred is set """
943+ for id in template.supported_ids:
944+ if id not in self.supported_ids:
945+ if self.debug:
946+ print "merged code %s (id %s) from branch '%s' from %s" % \
947+ (template.supported_ids[id]['ids'][0]['code'], id,
948+ template.supported_ids[id]['ids'][0]['origin'][0], template.filename)
949+ self.supported_ids[id] = template.supported_ids[id]
950+ else:
951+ for ent in template.supported_ids[id]['ids']:
952+ found = False
953+ for ent2 in self.supported_ids[id]['ids']:
954+ if ent2['code'] != ent['code']:
955+ continue
956+ found = True
957+ ent2['origin'].append(ent['origin'][0])
958+ if ent['test'] != ent2['test'] or \
959+ ent['desc'] != ent2['desc']:
960+ if newer_preferred:
961+ ent2['test'] = ent['test']
962+ ent2['desc'] = ent['desc']
963+ if not found:
964+ if self.debug:
965+ print "adding new ids code '%s' from branch '%s' for string id %s" % \
966+ (ent['code'], template.supported_ids[id]['ids'][0]['origin'][0], id)
967+ self.supported_ids[id]['ids'].append(ent)
968+
969+ def add_translation(self, lang, id, translation):
970+ if id not in self.supported_ids:
971+ if self.debug:
972+ print "*warn* obsolete string id %s for lang %s" % (id, lang)
973+ return
974+ self.supported_ids[id]['lang'][lang] = translation
975+
976+ def merge_translations(self, lang, xtb, master_xtb = None, newer_preferred = True):
977+ if lang not in self.supported_langs:
978+ self.supported_langs[lang] = xtb.filename
979+ for id in xtb.strings:
980+ if id not in self.supported_ids:
981+ # d'oh!! obsolete translation?
982+ self.update_stats(lang, obsolete = 1)
983+ continue
984+ if not self.is_string_valid_for_lang(id, lang):
985+ # string not wanted for that lang, skipped
986+ continue
987+ if 'lang' not in self.supported_ids[id]:
988+ self.supported_ids[id]['lang'] = {}
989+ if lang in self.supported_ids[id]['lang']:
990+ # already have a translation for this string
991+ if newer_preferred and xtb.strings[id] != self.supported_ids[id]['lang'][lang]:
992+ self.supported_ids[id]['lang'][lang] = xtb.strings[id]
993+ else:
994+ self.update_stats(lang, translated_upstream = 1)
995+ self.supported_ids[id]['lang'][lang] = xtb.strings[id]
996+ if master_xtb is not None:
997+ master_xtb.strings[id] = xtb.strings[id]
998+
999+ def read_string(self, node, test = None):
1000+ desc = node.getAttribute('desc')
1001+ name = node.getAttribute('name')
1002+ if not node.firstChild:
1003+ # no string? weird. Skip. (e.g. IDS_LOAD_STATE_IDLE)
1004+ return
1005+
1006+ # Get a/ the full string from the grd, b/ its transformation
1007+ # into the smaller version found in xtb files (val) and c/ another into
1008+ # something suitable for the 64bit key generator (kval)
1009+
1010+ orig_val = "".join([ n.toxml() for n in node.childNodes ])
1011+
1012+ # encode the value to create the 64bit ID needed for the xtb mapping.
1013+ #
1014+ # grd: 'f&amp;oo &quot;<ph name="IDS_xX">$1<ex>blabla</ex></ph>&quot; bar'
1015+ # xtb: 'f&amp;oo &quot;<ph name="IDS_XX"/>&quot; bar'
1016+ # but the string used to create the 64bit id is only 'f&oo "IDS_XX" bar'.
1017+ # Also, the final value must be positive, while FingerPrint() returns
1018+ # a signed long. Of course, none of this is documented...
1019+
1020+ # grd->xtb
1021+ for x in node.getElementsByTagName('ph'):
1022+ while x.hasChildNodes():
1023+ x.removeChild(x.childNodes[0])
1024+ val = "".join([ n.toxml() for n in node.childNodes ]).strip()
1025+ # xtb->id
1026+ kval = StringCvt().decode_xml_entities(unescape(self._PH_REGEXP.sub(self.uc, val))).encode('utf-8')
1027+ kval = kval.replace('&quot;', '"') # not replaced by unescape()
1028+
1029+ val = self._PH_REGEXP.sub(self.uc_name, val)
1030+ val = val.encode("ascii", "xmlcharrefreplace").strip().encode('utf-8')
1031+
1032+ # finally, create the 64bit ID
1033+ id = str(FingerPrint(kval) & 0x7fffffffffffffffL)
1034+
1035+ if val == '':
1036+ # unexpect <message/> block with attributes but without value, skip
1037+ return
1038+
1039+ if id not in self.supported_ids:
1040+ self.supported_ids[id] = { 'ids': [] }
1041+ self.supported_ids[id]['ids'].append({ 'code': name, 'desc': desc,
1042+ 'val': val, 'test': test,
1043+ 'origin': [ self.branch_name ] })
1044+
1045+ def read_strings(self, node, test = None):
1046+ for n in node.childNodes:
1047+ if n.nodeName == '#text' or n.nodeName == '#comment':
1048+ # comments, skip
1049+ continue
1050+ if n.nodeName == 'message':
1051+ self.read_string(n, test)
1052+ continue
1053+ if n.nodeName == 'if':
1054+ expr = n.getAttribute('expr')
1055+ if expr is not None and test is not None:
1056+ assert "nested <if> not supported"
1057+ self.read_strings(n, expr)
1058+ continue
1059+ if n.nodeName == 'part':
1060+ f = n.getAttribute('file')
1061+ qualified_file = os.path.join(os.path.dirname(self.filename), f)
1062+ self.import_file(override_filename=qualified_file)
1063+ continue
1064+ raise Exception("unknown tag (<%s> type %s): ''%s''" % \
1065+ (n.nodeName, n.nodeType, n.toxml()))
1066+
1067+ def import_json_file(self, filename):
1068+ # unlike its name seems to indicate, this file is definitely not a json file.
1069+ # It's a python object, dumped in a file. It means it's far easier to parse
1070+ # because there's no extra unescaping to do on all the strings. It also
1071+ # means we can't use the json module
1072+ rfile = os.path.join(self.branch_dir, filename)
1073+ if self.debug:
1074+ print "parse_json('%s') [%s]" % (filename, rfile)
1075+ fd = open(rfile, "rb")
1076+ data = fd.read()
1077+ fd.close()
1078+ vars = { '__builtins__': { 'True': True, 'False': False } } # prevent eval from using the real current globals
1079+ data = eval(data, vars)
1080+ # Check if this is a format we support
1081+ if 'policy_definitions' in data and len(data['policy_definitions']) > 0 and \
1082+ 'caption' not in data['policy_definitions'][0]:
1083+ # most probably Chromium v9. It used 'annotations' instead of 'caption'
1084+ # Not worth supporting that, all the strings we need in v9 are already in
1085+ # the grd file. Skip this json file
1086+ if self.debug:
1087+ print "Found older unsupported json format. Skipped"
1088+ return
1089+ if 'messages' in data:
1090+ for msg in data['messages']:
1091+ self.read_policy('IDS_POLICY_' + msg.upper(),
1092+ data['messages'][msg]['desc'],
1093+ data['messages'][msg]['text'])
1094+ if 'policy_definitions' in data:
1095+ for policy in data['policy_definitions']:
1096+ name = 'IDS_POLICY_' + policy['name'].upper()
1097+ if policy['type'] in [ 'main', 'int', 'string', 'list', 'string-enum', 'int-enum', 'string-enum-list' ]:
1098+ # caption
1099+ self.read_policy(name + '_CAPTION',
1100+ "Caption of the '%s' policy." % policy['name'],
1101+ policy['caption'])
1102+ # label (optional)
1103+ if 'label' in policy:
1104+ self.read_policy(name + '_LABEL',
1105+ "Label of the '%s' policy." % policy['name'],
1106+ policy['label'])
1107+ # desc
1108+ self.read_policy(name + '_DESC',
1109+ "Description of the '%s' policy." % policy['name'],
1110+ policy['desc'])
1111+ if policy['type'] in [ 'string-enum', 'int-enum', 'string-enum-list' ]:
1112+ for item in policy['items']:
1113+ self.read_policy('IDS_POLICY_ENUM_' + item['name'].upper().replace(' ', '_') + '_CAPTION',
1114+ "Label in a '%s' dropdown menu for selecting '%s'" % \
1115+ (policy['name'], item['name']),
1116+ item['caption'])
1117+ continue
1118+ if policy['type'] == 'group':
1119+ # group caption
1120+ self.read_policy(name + '_CAPTION',
1121+ "Caption of the group of '%s' related policies." % name,
1122+ policy['caption'])
1123+ # group label (optional)
1124+ if 'label' in policy:
1125+ self.read_policy(name + '_LABEL',
1126+ "Label of the group of '%s' related policies." % name,
1127+ policy['label'])
1128+ # group desc
1129+ self.read_policy(name + '_DESC',
1130+ "Description of the group of '%s' related policies." % name,
1131+ policy['desc'])
1132+ for spolicy in policy['policies']:
1133+ sname = 'IDS_POLICY_' + spolicy['name'].upper()
1134+ # desc
1135+ self.read_policy(sname + '_DESC',
1136+ "Description of the '%s' policy." % spolicy['name'],
1137+ spolicy['desc'])
1138+ # label (optional)
1139+ if 'label' in spolicy:
1140+ self.read_policy(sname + '_LABEL',
1141+ "Label of the '%s' policy." % spolicy['name'],
1142+ spolicy['label'])
1143+ # caption
1144+ self.read_policy(sname + '_CAPTION',
1145+ "Caption of the '%s' policy." % spolicy['name'],
1146+ spolicy['caption'])
1147+ if spolicy['type'] in [ 'int-enum', 'string-enum' ]:
1148+ # only caption
1149+ for item in spolicy['items']:
1150+ self.read_policy('IDS_POLICY_ENUM_' + item['name'].upper() + '_CAPTION',
1151+ "Label in a '%s' dropdown menu for selecting a '%s' of '%s'" % \
1152+ (policy['name'], spolicy['name'], item['name']),
1153+ item['caption'])
1154+ continue
1155+ # The new type is not yet being used: http://code.google.com/p/chromium/issues/detail?id=108992
1156+ if policy['type'] == 'external':
1157+ continue
1158+ if policy['type'] == 'dict':
1159+ continue
1160+
1161+ assert False, "Policy type '%s' not supported while parsing %s" % (policy['type'], rfile)
1162+
1163+ def read_policy(self, name, desc, text):
1164+ xml = '<x><message name="%s" desc="%s">\n%s\n</message></x>' % (name, desc, text)
1165+ dom = minidom.parseString(xml)
1166+ self.read_strings(dom.getElementsByTagName('x')[0])
1167+
1168+ def _add_xtb(self, node):
1169+ if node.nodeName != 'file':
1170+ return
1171+ path = node.getAttribute('path')
1172+ m = re.match(r'.*_([^_]+)\.xtb', path)
1173+ flang = m.group(1)
1174+ lang = node.getAttribute('lang')
1175+ tlang = lang
1176+ if self.lang_mapping is not None and lang in self.lang_mapping:
1177+ if self.debug:
1178+ print "# mapping lang '%s' to '%s'" % (lang, self.lang_mapping[lang])
1179+ tlang = self.lang_mapping[lang]
1180+ tlang = tlang.replace('-', '_')
1181+ self.supported_langs[lang] = os.path.normpath(os.path.join(os.path.dirname(self.filename), path))
1182+ self.translated_strings[lang] = {}
1183+ glang = lang
1184+ if flang == 'iw':
1185+ glang = flang
1186+ #assert lang not in self.mapped_langs, "'%s' already in self.mapped_langs" % lang
1187+ if lang not in self.mapped_langs:
1188+ self.mapped_langs[lang] = { 'xtb_file': flang, 'grit': glang, 'gettext': tlang }
1189+
1190+ def import_file(self, override_filename=None):
1191+ if override_filename:
1192+ assert self.branch_dir
1193+ filename = os.path.join(self.branch_dir, override_filename)
1194+ else:
1195+ filename = os.path.join(self.branch_dir, self.filename) if self.branch_dir is not None \
1196+ else self.filename
1197+ self.supported_langs = {}
1198+ self.mtime = self.get_mtime(self.filename)
1199+ if self.debug:
1200+ print "minidom.parse(%s)" % filename
1201+ dom = minidom.parse(filename)
1202+ grits = dom.getElementsByTagName('grit')
1203+ if not grits:
1204+ grits = dom.getElementsByTagName('grit-part')
1205+ grit = grits[0]
1206+ for node in grit.childNodes:
1207+ if node.nodeName == '#text' or node.nodeName == '#comment':
1208+ # comments, skip
1209+ continue
1210+ if node.nodeName == 'outputs':
1211+ # skip, nothing for us here
1212+ continue
1213+ if node.nodeName == 'translations':
1214+ # collect the supported langs by scanning the list of xtb files
1215+ for n in node.childNodes:
1216+ if n.nodeName == 'if':
1217+ for nn in n.childNodes:
1218+ self._add_xtb(nn)
1219+ continue
1220+ self._add_xtb(n)
1221+ continue
1222+ if node.nodeName == 'release':
1223+ for n in node.childNodes:
1224+ if n.nodeName == '#text' or n.nodeName == '#comment':
1225+ # comments, skip
1226+ continue
1227+ if n.nodeName == 'includes':
1228+ # skip, nothing for us here
1229+ continue
1230+ if n.nodeName == 'structures':
1231+ for sn in n.childNodes:
1232+ if sn.nodeName != 'structure':
1233+ continue
1234+ type = sn.getAttribute('type')
1235+ if type == 'dialog':
1236+ # nothing for us here
1237+ continue
1238+ name = sn.getAttribute('name')
1239+ file = sn.getAttribute('file')
1240+ if type == 'policy_template_metafile':
1241+ # included file containing the strings that are usually in the <messages> tree.
1242+ fname = os.path.normpath(os.path.join(os.path.dirname(self.filename), file))
1243+ self.import_json_file(fname)
1244+ continue
1245+ else:
1246+ if self.debug:
1247+ print "unknown <structure> type found ('%s') in %s" % (type, self.filename)
1248+ continue
1249+ if n.nodeName == 'messages':
1250+ self.read_strings(n)
1251+ continue
1252+ print "unknown tag (<%s> type %s): ''%s''" % (n.nodeName, n.nodeType, n.toxml())
1253+ continue
1254+ print "unknown tag (<%s> type %s): ''%s''" % (node.nodeName, node.nodeType, node.toxml())
1255+
1256+class XtbFile(PoFile):
1257+ """
1258+ Read and write a Grit XTB file
1259+ """
1260+
1261+ def __init__(self, lang, filename, grd, date = None, debug = None,
1262+ branch_name = "default", branch_dir = os.getcwd()):
1263+ super(XtbFile, self).__init__(lang, filename, grd, date = date, debug = debug,
1264+ branch_name = branch_name, branch_dir = branch_dir)
1265+ self.template = grd
1266+ self.strings = {}
1267+ self.strings_updated = 0
1268+ self.strings_new = 0
1269+ self.strings_order = [] # needed to recreate xtb files in a similar order :(
1270+
1271+ def add_translation(self, id, string):
1272+ assert id in self.template.supported_ids, "'%s' is not in supported_ids (file=%s)" % (id, self.filename)
1273+ while string[-1:] == '\n' and self.template.supported_ids[id]['ids'][0]['val'][-1:] != '\n':
1274+ # prevent the `msgid' and `msgstr' entries do not both end with '\n' error
1275+ if self.debug:
1276+ print "Found unwanted \\n at the end of translation id " + id + " lang " + self.lang + ". Dropped"
1277+ string = string[:-1]
1278+ while string[0] == '\n' and self.template.supported_ids[id]['ids'][0]['val'][0] != '\n':
1279+ # prevent the `msgid' and `msgstr' entries do not both begin with '\n' error
1280+ if self.debug:
1281+ print "Found unwanted \\n at the begin of translation id " + id + " lang " + self.lang + ". Dropped"
1282+ string = string[1:]
1283+ self.strings[id] = string
1284+ self.strings_order.append(id)
1285+
1286+ def write_header(self):
1287+ self.write('<?xml version="1.0" ?>\n')
1288+ self.write('<!DOCTYPE translationbundle>\n')
1289+ self.write('<translationbundle lang="%s">\n' % \
1290+ self.template.mapped_langs[self.lang]['grit'])
1291+
1292+ def write_footer(self):
1293+ self.write('</translationbundle>')
1294+
1295+ def write_all_strings(self):
1296+ for id in self.strings_order:
1297+ if id in self.strings:
1298+ self.write('<translation id="%s">%s</translation>\n' % \
1299+ (id, self.strings[id]))
1300+ for id in sorted(self.strings.keys()):
1301+ if id in self.strings_order:
1302+ continue
1303+ self.write('<translation id="%s">%s</translation>\n' % \
1304+ (id, self.strings[id]))
1305+
1306+ def import_po(self, po):
1307+ for string in po.strings:
1308+ if string['string'] == '':
1309+ continue
1310+ self.add_string(string['id'], string['extracted'],
1311+ string['string'], string['translation'])
1312+
1313+ def import_file(self):
1314+ self.open()
1315+ file = self.fd.read() # *sigh*
1316+ self.close()
1317+ imported = 0
1318+ for m in re.finditer('<translation id="(.*?)">(.*?)</translation>',
1319+ file, re.S):
1320+ if m.group(1) not in self.template.supported_ids:
1321+ if self.debug:
1322+ print "found a translation for obsolete string id %s in upstream xtb %s" % (m.group(1), self.filename)
1323+ continue
1324+ self.add_translation(m.group(1), m.group(2))
1325+ imported += 1
1326+ for m in re.finditer('<translationbundle lang="(.*?)">', file):
1327+ lang = m.group(1)
1328+ if self.lang in self.template.mapped_langs:
1329+ assert self.template.mapped_langs[self.lang]['grit'] == lang, \
1330+ "bad lang mapping for '%s' while importing %s, expected '%s'" % \
1331+ (lang, self.filename, self.template.mapped_langs[self.lang]['grit'])
1332+ else:
1333+ tlang = lang
1334+ if self.template.lang_mapping is not None and lang in self.template.lang_mapping:
1335+ if self.debug:
1336+ print "# mapping lang '%s' to '%s'" % (lang, self.template.lang_mapping[lang])
1337+ tlang = self.template.lang_mapping[lang]
1338+ tlang = tlang.replace('-', '_')
1339+ flang = lang.replace('@', '-')
1340+ self.template.mapped_langs[lang] = { 'xtb_file': flang, 'grit': lang, 'gettext': tlang }
1341+ if self.debug:
1342+ print "imported %d strings from the xtb file into lang '%s'" % (imported, self.lang)
1343+ self.mtime = self.get_mtime(self.filename)
1344+
1345+###
1346+
1347+class Converter(dict):
1348+ """
1349+ Given a grd template and its xtb translations,
1350+ a/ exports gettext pot template and po translations,
1351+ possibly by merging grd/xtb files from multiple branches
1352+ or
1353+ b/ imports and merges some gettext po translations,
1354+ and exports xtb translations
1355+ """
1356+
1357+ def __init__(self, template_filename, lang_mapping = None, date = None, debug = False,
1358+ template_mapping = {}, html_output = False, branches = None):
1359+ self.debug = debug
1360+ self.translations = {}
1361+ self.errors = 0
1362+ self.template_changes = 0
1363+ self.translations_changes = 0
1364+ self.lang_mapping = lang_mapping
1365+ self.template_mapping = template_mapping
1366+ self.file_mapping = {}
1367+ self.html_output = html_output
1368+ self.stats = {}
1369+ self.branches = branches if branches is not None else [ { 'branch': 'default', 'dir': os.getcwd(), 'grd': template_filename } ]
1370+
1371+ # read a grd template from a file
1372+ self.template = GrdFile(self.branches[0]['grd'], date, lang_mapping = self.lang_mapping, debug = self.debug,
1373+ branch_name = self.branches[0]['branch'], branch_dir = self.branches[0]['dir'])
1374+ self.file_mapping['grd'] = { 'src': self.branches[0]['grd'],
1375+ 'branches': { self.branches[0]['branch']: self.branches[0]['dir'] } }
1376+ if 'mapped_grd' in self.branches[0]:
1377+ self.file_mapping['grd']['mapped_grd'] = self.branches[0]['mapped_grd']
1378+ self.template.import_file()
1379+ self.template_pot = None
1380+ for lang, file in zip(self.template.get_supported_langs(),
1381+ self.template.get_supported_lang_filenames()):
1382+ try:
1383+ # also read all the xtb files referenced by this grd template
1384+ rfile = os.path.join(self.branches[0]['dir'] , file)
1385+ xtb = XtbFile(lang, file, self.template, date = self.template.get_mtime(file), debug = self.debug,
1386+ branch_name = self.branches[0]['branch'], branch_dir = self.branches[0]['dir'])
1387+ xtb.import_file()
1388+
1389+ self.file_mapping['lang_' + lang] = { 'src': file,
1390+ 'branches': { self.branches[0]['branch']: self.branches[0]['dir'] } }
1391+ self.stats[lang] = { 'strings': self.template.get_supported_strings_count(lang),
1392+ 'translated_upstream': 0,
1393+ 'changed_in_gettext': 0,
1394+ 'rejected': 0
1395+ }
1396+ self.template.merge_translations(lang, xtb)
1397+ self.translations[lang] = xtb
1398+ except Exception, e:
1399+ print "Skipping a XTB file, ERROR while importing xtb %s from grd file %s: %s" % (file, self.branches[0]['grd'], str(e))
1400+
1401+ # read other grd templates
1402+ if len(self.branches) > 1:
1403+ for branch in self.branches[1:]:
1404+ if self.debug:
1405+ print "merging %s from branch '%s' from %s" % (branch['grd'], branch['branch'], branch['dir'])
1406+ template = GrdFile(branch['grd'], date, lang_mapping = self.lang_mapping, debug = self.debug,
1407+ branch_name = branch['branch'], branch_dir = branch['dir'])
1408+ self.file_mapping['grd']['branches'][branch['branch']] = branch['dir']
1409+ template.import_file()
1410+ self.template.merge_template(template, newer_preferred = False)
1411+ for lang, file in zip(template.get_supported_langs(),
1412+ template.get_supported_lang_filenames()):
1413+ xtb = XtbFile(lang, file, self.template, date = template.get_mtime(file), debug = self.debug,
1414+ branch_name = branch['branch'], branch_dir = branch['dir'])
1415+ if 'lang_' + lang not in self.file_mapping:
1416+ self.file_mapping['lang_' + lang] = { 'src': file, 'branches': {} }
1417+ self.file_mapping['lang_' + lang]['branches'][branch['branch']] = branch['dir']
1418+ # TODO: stats
1419+ xtb.import_file()
1420+ if lang not in self.translations:
1421+ if self.debug:
1422+ print "Add lang '%s' as master xtb for alt branch '%s'" % (lang, branch['branch'])
1423+ self.translations[lang] = xtb
1424+ self.template.merge_translations(lang, xtb, master_xtb = self.translations[lang],
1425+ newer_preferred = False)
1426+
1427+ def export_gettext_files(self, directory):
1428+ fname = self.file_mapping['grd']['mapped_grd'] \
1429+ if 'mapped_grd' in self.file_mapping['grd'] else self.template.filename
1430+ name = os.path.splitext(os.path.basename(fname))[0]
1431+ if directory is not None:
1432+ directory = os.path.join(directory, name)
1433+ if not os.path.isdir(directory):
1434+ os.makedirs(directory, 0755)
1435+ filename = os.path.join(directory, name + ".pot")
1436+ else:
1437+ filename = os.path.splitext(fname)[0] + ".pot"
1438+ # create a pot template and merge the grd strings into it
1439+ self.template_pot = PotFile(filename, date = self.template.mtime, debug = self.debug)
1440+ self.template_pot.import_grd(self.template)
1441+ # write it to a file
1442+ self.template_changes += self.template_pot.export_file(directory = directory)
1443+
1444+ # do the same for all langs (xtb -> po)
1445+ for lang in self.translations:
1446+ gtlang = self.template.mapped_langs[lang]['gettext']
1447+ file = os.path.join(os.path.dirname(filename), gtlang + ".po")
1448+ po = PoFile(gtlang, file, self.template_pot,
1449+ date = self.translations[lang].translation_date, debug = self.debug)
1450+ po.import_xtb(self.translations[lang])
1451+ self.translations_changes += po.export_file(directory)
1452+
1453+ def export_grit_xtb_file(self, lang, directory):
1454+ name = os.path.splitext(os.path.basename(self.template.filename))[0]
1455+ file = os.path.join(directory, os.path.basename(self.template.supported_langs[lang]))
1456+ if len(self.translations[lang].strings.keys()) > 0:
1457+ if 'lang_' + lang in self.file_mapping:
1458+ self.file_mapping['lang_' + lang]['dst'] = file
1459+ else:
1460+ self.file_mapping['lang_' + lang] = { 'src': None, 'dst': file }
1461+ self.translations[lang].export_file(filename = file)
1462+
1463+ def export_grit_files(self, directory, langs):
1464+ grd_dst = os.path.join(directory, os.path.basename(self.template.filename))
1465+ if len(self.translations.keys()) == 0:
1466+ if self.debug:
1467+ print "no translation at all, nothing to export here (template: %s)" % self.template.filename
1468+ return
1469+ if not os.path.isdir(directory):
1470+ os.makedirs(directory, 0755)
1471+ # 'langs' may contain langs for which this template no longer have translations for.
1472+ # They need to be dropped from the grd file
1473+ self.template.export_file(filename = grd_dst, global_langs = langs, langs = self.translations.keys())
1474+ self.file_mapping['grd']['dst'] = grd_dst
1475+ self.file_mapping['grd']['dir'] = directory[:-len(os.path.dirname(self.template.filename)) - 1]
1476+ for lang in self.translations:
1477+ prefix = self.template.supported_langs[lang]
1478+ fdirectory = os.path.normpath(os.path.join(self.file_mapping['grd']['dir'], os.path.dirname(prefix)))
1479+ if not os.path.isdir(fdirectory):
1480+ os.makedirs(fdirectory, 0755)
1481+ self.export_grit_xtb_file(lang, fdirectory)
1482+
1483+ def get_supported_strings_count(self):
1484+ return len(self.template.supported_ids.keys())
1485+
1486+ def compare_translations(self, old, new, id, lang):
1487+ # strip leading and trailing whitespaces from the upstream strings
1488+ # (this should be done upstream)
1489+ old = old.strip()
1490+ if old != new:
1491+ s = self.template.supported_ids[id]['ids'][0]['val'] if 'ids' in self.template.supported_ids[id] else "<none?>"
1492+ if self.debug:
1493+ print "Found a different translation for id %s in lang '%s':\n string: \"%s\"\n " \
1494+ "upstream: \"%s\"\n launchpad: \"%s\"\n" % (id, lang, s, old, new)
1495+ return old == new
1496+
1497+ def import_gettext_po_file(self, lang, filename):
1498+ """ import a single lang file into the current translations set,
1499+ matching the current template. Could be useful to merge the upstream
1500+ and launchpad translations, or to merge strings from another project
1501+ (like webkit) """
1502+ po = PoFile(self.template.mapped_langs[lang]['gettext'], filename, self.template,
1503+ date = self.template.get_mtime(filename), debug = self.debug)
1504+ po.import_file()
1505+ # no need to continue if there are no translation in this po
1506+ translated_count = 0
1507+ for s in po.strings:
1508+ if s['string'] != '""' and s['translation'] != '""':
1509+ translated_count += 1
1510+ if translated_count == 0:
1511+ if self.debug:
1512+ print "No translation found for lang %s in %s" % (lang, filename)
1513+ return
1514+ if lang not in self.translations:
1515+ # assuming the filename should be third_party/launchpad_translations/<template_name>_<lang>.xtb
1516+ # (relative to $SRC), we need it relatively to the grd directory
1517+ tname = os.path.splitext(os.path.basename(self.template.filename))[0]
1518+ f = os.path.normpath(os.path.join('third_party/launchpad_translations',
1519+ tname + '_' + self.template.mapped_langs[lang]['xtb_file'] + '.xtb'))
1520+ self.translations[lang] = XtbFile(lang, f, self.template, date = po.mtime, debug = self.debug)
1521+ self.template.supported_langs[lang] = f # *sigh*
1522+
1523+ lp669831_skipped = 0
1524+ for string in po.strings:
1525+ if 'id' not in string:
1526+ continue # PO header
1527+ id = string['id']
1528+ if id in self.template.supported_ids:
1529+ if 'conditions' in string:
1530+ # test the lang against all those conditions. If at least one passes, we need
1531+ # the string
1532+ found = False
1533+ for c in string['conditions']:
1534+ found |= EvalConditions().lang_eval(c, lang)
1535+ if found is False:
1536+ self.template.update_stats(lang, skipped_lang = 1)
1537+ if self.debug:
1538+ print "Skipped string (lang condition) for %s/%s: %s" % \
1539+ (os.path.splitext(os.path.basename(self.template.filename))[0],
1540+ lang, repr(string))
1541+ continue
1542+ # workaround bug https://bugs.launchpad.net/rosetta/+bug/669831
1543+ ustring = StringCvt().gettext2xtb(string['string'])
1544+ gt_translation = string['translation'][1:-1].replace('"\n"', '')
1545+ string['translation'] = StringCvt().gettext2xtb(string['translation'])
1546+ if string['translation'] != "":
1547+ while string['translation'][-1:] == '\n' and ustring[-1:] != '\n':
1548+ # prevent the `msgid' and `msgstr' entries do not both end with '\n' error
1549+ if self.debug:
1550+ print "Found unwanted \\n at the end of translation id " + id + " lang " + self.lang + ". Dropped"
1551+ string['translation'] = string['translation'][:-1]
1552+ while string['translation'][0] == '\n' and ustring[0] != '\n':
1553+ # prevent the `msgid' and `msgstr' entries do not both begin with '\n' error
1554+ if self.debug:
1555+ print "Found unwanted \\n at the begin of translation id " + id + " lang " + self.lang + ". Dropped"
1556+ string['translation'] = string['translation'][1:]
1557+ grit_str = StringCvt().decode_xml_entities(self.template.supported_ids[id]['ids'][0]['val'])
1558+ if False and 'ids' in self.template.supported_ids[id] and \
1559+ ustring != grit_str:
1560+ # the string for this id is no longer the same, skip it
1561+ lp669831_skipped += 1
1562+ if self.debug:
1563+ print "lp669831_skipped:\n lp: '%s'\n grd: '%s'" % (ustring, grit_str)
1564+ continue
1565+ # check for xml errors when '<' or '>' are in the string
1566+ if string['translation'].find('<') >= 0 or \
1567+ string['translation'].find('>') >= 0:
1568+ try:
1569+ # try to parse it with minidom (it's slow!!), and skip if it fails
1570+ s = u"<x>" + string['translation'] + u"</x>"
1571+ dom = minidom.parseString(s.encode('utf-8'))
1572+ except Exception as inst:
1573+ print "Parse error in '%s/%s' for id %s. Skipped.\n%s\n%s" % \
1574+ (os.path.splitext(os.path.basename(self.template.filename))[0], lang, id,
1575+ repr(string['translation']), inst)
1576+ continue
1577+ # if the upstream string is not empty, but the contributed string is, keep
1578+ # the upstream string untouched
1579+ if string['translation'] == '':
1580+ continue
1581+ # check if we have the same variables in both the upstream string and its
1582+ # translation. Otherwise, complain and reject the translation
1583+ if 'ids' in self.template.supported_ids[id]:
1584+ uvars = sorted([e for e in re.split('(<ph name=".*?"/>)', self.template.supported_ids[id]['ids'][0]['val']) \
1585+ if re.match('^<ph name=".*?"/>$', e)])
1586+ tvars = sorted([e for e in re.split('(<ph name=".*?"/>)', string['translation'])\
1587+ if re.match('^<ph name=".*?"/>$', e)])
1588+ lostvars = list(set(uvars).difference(set(tvars)))
1589+ createdvars = list(set(tvars).difference(set(uvars)))
1590+ if len(lostvars) or len(createdvars):
1591+ template = os.path.splitext(os.path.basename(self.template.filename))[0].replace('_', '-')
1592+ self.errors += 1
1593+ if self.html_output:
1594+ print "<div class='error'>[<a id='pherr-%s-%d' href='javascript:toggle(\"pherr-%s-%d\");'>+</a>] " \
1595+ "<b>ERROR</b>: Found mismatching placeholder variables in string id %s of <b>%s</b> lang <b>%s</b>" % \
1596+ (template, self.errors, template, self.errors, id, template, lang)
1597+ else:
1598+ print "ERROR: Found mismatching placeholder variables in string id %s of %s/%s:" % \
1599+ (id, template, lang)
1600+ url = 'https://translations.launchpad.net/chromium-browser/translations/+pots/%s/%s/+translate?batch=10&show=all&search=%s' % \
1601+ (template, self.template.mapped_langs[lang]['gettext'], urllib.quote(gt_translation.encode('utf-8')))
1602+ if self.html_output:
1603+ print "<div id='pherr-%s-%d-t' style='display: none'>\n" \
1604+ "<fieldset><legend>Details</legend><p><ul>" % (template, self.errors)
1605+ print "<li> <a href='%s'>this string in Launchpad</a>\n" % url
1606+ if len(lostvars):
1607+ print " <li> expected but not found: <code>%s</code>" % " ".join([ re.sub(r'<ph name="(.*?)"/>', r'%{\1}', s) for s in lostvars ])
1608+ if len(createdvars):
1609+ print " <li> found but not expected: <code>%s</code>" % " ".join([ re.sub(r'<ph name="(.*?)"/>', r'%{\1}', s) for s in createdvars ])
1610+ print "</ul><table border='1'>" \
1611+ "<tr><th rowspan='2'>GetText</th><th>template</th><td><code>%s</code></td></tr>\n" \
1612+ "<tr><th>translation</th><td><code>%s</code></td></tr>\n" \
1613+ "<tr><th rowspan='2'>Grit</th><th>template</th><td><code>%s</code></td></tr>\n" \
1614+ "<tr><th>translation</th><td><code>%s</code></td></tr>\n" \
1615+ "</table><p> => <b>translation skipped</b>\n" % \
1616+ (string['string'][1:-1].replace('"\n"', '').replace('<', '&lt;').replace('>', '&gt;'),
1617+ gt_translation.replace('<', '&lt;').replace('>', '&gt;'),
1618+ self.template.supported_ids[id]['ids'][0]['val'].replace('<', '&lt;').replace('>', '&gt;'),
1619+ string['translation'].replace('<', '&lt;').replace('>', '&gt;'))
1620+ print "</fieldset></div></div>"
1621+ else:
1622+ if len(lostvars):
1623+ print " - expected but not found: " + " ".join(lostvars)
1624+ if len(createdvars):
1625+ print " - found but not expected: " + " ".join(createdvars)
1626+ print " string: '%s'\n translation: '%s'\n gettext: '%s'\n url: %s\n => translation skipped\n" % \
1627+ (self.template.supported_ids[id]['ids'][0]['val'], string['translation'], gt_translation, url)
1628+ continue
1629+ # check if the translated string is the same
1630+ if 'lang' in self.template.supported_ids[id] and \
1631+ lang in self.template.supported_ids[id]['lang']:
1632+ # compare
1633+ if self.compare_translations(self.template.supported_ids[id]['lang'][lang],
1634+ string['translation'], id, lang):
1635+ continue # it's the same
1636+ if id in self.translations[lang].strings:
1637+ # already added from a previously merged gettext po file
1638+ if self.debug:
1639+ print "already added from a previously merged gettext po file for" + \
1640+ " template %s %s id %s in lang %s: %s" % \
1641+ (self.template.branch_name, self.template.filename,
1642+ id, lang, repr(string['translation']))
1643+ # compare
1644+ if self.compare_translations(self.translations[lang].strings[id],
1645+ string['translation'], id, lang):
1646+ continue # it's the same
1647+ # update it..
1648+ if self.debug:
1649+ print "updated string for template %s %s id %s in lang %s: %s" % \
1650+ (self.template.branch_name, self.template.filename, id, lang,
1651+ repr(string['translation']))
1652+ self.template.update_stats(lang, updated = 1)
1653+ self.translations[lang].strings[id] = string['translation']
1654+ self.translations[lang].strings_updated += 1
1655+ elif id in self.translations[lang].strings:
1656+ # already added from a previously merged gettext po file
1657+ if self.debug:
1658+ print "already added from a previously merged gettext po file for" + \
1659+ "template %s %s id %s in lang %s: %s" % \
1660+ (self.template.branch_name, self.template.filename,
1661+ id, lang, repr(string['translation']))
1662+ # compare
1663+ if self.compare_translations(self.translations[lang].strings[id],
1664+ string['translation'], id, lang):
1665+ continue # it's the same
1666+ # update it..
1667+ self.translations[lang].strings[id] = string['translation']
1668+ else:
1669+ # add
1670+ if self.debug:
1671+ print "add new string for template %s %s id %s in lang %s: %s" % \
1672+ (self.template.branch_name, self.template.filename,
1673+ id, lang, repr(string['translation']))
1674+ self.template.update_stats(lang, new = 1)
1675+ self.translations[lang].strings[id] = string['translation']
1676+ self.translations[lang].strings_new += 1
1677+ if self.debug and lp669831_skipped > 0:
1678+ print "lp669831: skipped %s bogus/obsolete strings from %s" % \
1679+ (lp669831_skipped, filename[filename[:filename.rfind('/')].rfind('/') + 1:])
1680+
1681+ def import_gettext_po_files(self, directory):
1682+ fname = self.file_mapping['grd']['mapped_grd'] \
1683+ if 'mapped_grd' in self.file_mapping['grd'] else self.template.filename
1684+ template_name = os.path.splitext(os.path.basename(fname))[0]
1685+ directory = os.path.join(directory, template_name)
1686+ if not os.path.isdir(directory):
1687+ if self.debug:
1688+ print "WARN: Launchpad didn't export anything for template '%s' [%s]" % (template_name, directory)
1689+ return
1690+ for file in os.listdir(directory):
1691+ base, ext = os.path.splitext(file)
1692+ if ext != ".po":
1693+ continue
1694+ # 'base' is a gettext lang, map it
1695+ lang = None
1696+ for l in self.template.mapped_langs:
1697+ if base == self.template.mapped_langs[l]['gettext']:
1698+ lang = l
1699+ break
1700+ if lang is None: # most probably a new lang, map back
1701+ lang = base.replace('_', '-')
1702+ for l in self.lang_mapping:
1703+ if lang == self.lang_mapping[l]:
1704+ lang = l
1705+ break
1706+ flang = lang.replace('@', '-')
1707+ self.template.mapped_langs[lang] = { 'xtb_file': flang, 'grit': lang, 'gettext': base }
1708+ self.import_gettext_po_file(lang, os.path.join(directory, file))
1709+ # remove from the supported langs list all langs with no translated strings
1710+ # (to catch either empty 'po' files exported by Launchpad, or 'po' files
1711+ # containing only obsolete or too new strings for this branch)
1712+ dropped = []
1713+ for lang in self.translations:
1714+ if len(self.translations[lang].strings.keys()) == 0:
1715+ if self.debug:
1716+ print "no translation found for template '%s' and lang '%s'. lang removed from the supported lang list" % \
1717+ (os.path.splitext(os.path.basename(self.template.filename))[0], lang)
1718+ del(self.template.supported_langs[lang])
1719+ dropped.append(lang)
1720+ for lang in dropped:
1721+ del(self.translations[lang])
1722+
1723+ def copy_grit_files(self, directory):
1724+ fname = self.file_mapping['grd']['mapped_grd'] \
1725+ if 'mapped_grd' in self.file_mapping['grd'] else self.template.filename
1726+ dst = os.path.join(directory, os.path.dirname(fname))
1727+ if not os.path.isdir(dst):
1728+ os.makedirs(dst, 0755)
1729+ shutil.copy2(fname, dst)
1730+ for lang in self.template.supported_langs:
1731+ dst = os.path.join(directory, os.path.dirname(self.translations[lang].filename))
1732+ if not os.path.isdir(dst):
1733+ os.makedirs(dst, 0755)
1734+ shutil.copy2(self.translations[lang].filename, dst)
1735+
1736+ def create_patches(self, directory):
1737+ if not os.path.isdir(directory):
1738+ os.makedirs(directory, 0755)
1739+ template_name = os.path.splitext(os.path.basename(self.template.filename))[0]
1740+ patch = codecs.open(os.path.join(directory, "translations-" + template_name + ".patch"),
1741+ "wb", encoding="utf-8")
1742+ for e in sorted(self.file_mapping.keys()):
1743+ if 'dst' not in self.file_mapping[e]:
1744+ self.file_mapping[e]['dst'] = None
1745+ if self.file_mapping[e]['src'] is not None and \
1746+ self.file_mapping[e]['dst'] is not None and \
1747+ filecmp.cmp(self.file_mapping[e]['src'], self.file_mapping[e]['dst']) == True:
1748+ continue # files are the same
1749+
1750+ if self.file_mapping[e]['src'] is not None:
1751+ fromfile = "old/" + self.file_mapping[e]['src']
1752+ tofile = "new/" + self.file_mapping[e]['src']
1753+ fromdate = datetime.fromtimestamp(self.template.get_mtime(
1754+ self.file_mapping[e]['src'])).strftime("%Y-%m-%d %H:%M:%S.%f000 +0000")
1755+ fromlines = codecs.open(self.file_mapping[e]['src'], 'rb', encoding="utf-8").readlines()
1756+ else:
1757+ fromfile = "old/" + self.file_mapping[e]['dst'][len(self.file_mapping['grd']['dir']) + 1:]
1758+ tofile = "new/" + self.file_mapping[e]['dst'][len(self.file_mapping['grd']['dir']) + 1:]
1759+ fromdate = datetime.fromtimestamp(0).strftime("%Y-%m-%d %H:%M:%S.%f000 +0000")
1760+ fromlines = ""
1761+ if self.file_mapping[e]['dst'] is not None:
1762+ todate = datetime.fromtimestamp(self.template.get_mtime(
1763+ self.file_mapping[e]['dst'])).strftime("%Y-%m-%d %H:%M:%S.%f000 +0000")
1764+ tolines = codecs.open(self.file_mapping[e]['dst'], 'rb', encoding="utf-8").readlines()
1765+ else:
1766+ todate = datetime.fromtimestamp(0).strftime("%Y-%m-%d %H:%M:%S.%f000 +0000")
1767+ tolines = ""
1768+ diff = unified_diff(fromlines, tolines, fromfile, tofile,
1769+ fromdate, todate, n=3)
1770+ patch.write("diff -Nur %s %s\n" % (fromfile, tofile))
1771+ s = ''.join(diff)
1772+ # fix the diff so that older patch (<< 2.6) don't fail on new files
1773+ s = re.sub(r'@@ -1,0 ', '@@ -0,0 ', s)
1774+ # ..and make sure patch is able to detect a patch removing files
1775+ s = re.sub(r'(@@ \S+) \+1,0 @@', '\\1 +0,0 @@', s)
1776+ patch.writelines(s)
1777+ if s[-1:] != '\n':
1778+ patch.write("\n\\ No newline at end of file\n")
1779+ patch.close()
1780+
1781+ def update_supported_langs_in_grd(self, grd_in, grd_out, langs):
1782+ fdi = codecs.open(grd_in, 'rb', encoding="utf-8")
1783+ fdo = codecs.open(grd_out, 'wb', encoding="utf-8")
1784+ # can't use minidom here as the file is manually generated and the
1785+ # output will create big diffs. parse the source file line by line
1786+ # and insert new langs in the <outputs> section (with type="data_package"
1787+ # or type="js_map_format"). Let everything else untouched
1788+ # FIXME: this is mostly a copy of GrdFile::export_file()
1789+ pak_found = False
1790+ pak_saved = []
1791+ has_ifs = False
1792+ for line in fdi.readlines():
1793+ if re.match(r'.*?<output filename=".*?" type="(data_package|js_map_format)"', line):
1794+ pak_found = True
1795+ pak_saved.append(line)
1796+ continue
1797+ if line.find('</outputs>') > 0:
1798+ pak_found = False
1799+ ours = langs[:]
1800+ chunks = {}
1801+ c = None
1802+ pak_if = None
1803+ pak_is_in_if = False
1804+ for l in pak_saved:
1805+ if l.find("<!-- ") > 0:
1806+ c = l
1807+ continue
1808+ if l.find("<if ") > -1:
1809+ c = l if c is None else c + l
1810+ has_ifs = True
1811+ pak_is_in_if = True
1812+ continue
1813+ if l.find("</if>") > -1:
1814+ c = l if c is None else c + l
1815+ pak_is_in_if = False
1816+ continue
1817+ m = re.match(r'.*?<output filename="(.*?)_([^_\.]+)\.(pak|js)" type="(data_package|js_map_format)" lang="(.*?)" />', l)
1818+ if m is not None:
1819+ x = { 'name': m.group(1), 'ext': m.group(3), 'lang': m.group(5), 'file_lang': m.group(2),
1820+ 'type': m.group(4), 'in_if': pak_is_in_if, 'line': l }
1821+ if c is not None:
1822+ x['comment'] = c
1823+ c = None
1824+ k = m.group(2) if m.group(2) != 'nb' else 'no'
1825+ chunks[k] = x
1826+ else:
1827+ if c is None:
1828+ c = l
1829+ else:
1830+ c += l
1831+ is_in_if = False
1832+ for lang in sorted(chunks.keys()):
1833+ tlang = lang if lang != 'no' else 'nb'
1834+ while len(ours) > 0 and ((ours[0] == 'nb' and 'no' < tlang) or (ours[0] != 'nb' and ours[0] < tlang)):
1835+ if ours[0] in chunks:
1836+ ours = ours[1:]
1837+ continue
1838+ if has_ifs and is_in_if is False:
1839+ fdo.write(' <if expr="pp_ifdef(\'use_third_party_translations\')">\n')
1840+ f = "%s_%s.%s" % (chunks[lang]['name'], ours[0], chunks[lang]['ext'])
1841+ fdo.write(' %s<output filename="%s" type="%s" lang="%s" />\n' % \
1842+ (' ' if has_ifs else '', f, chunks[lang]['type'], ours[0]))
1843+ is_in_if = True
1844+ if has_ifs and chunks[lang]['in_if'] is False:
1845+ if 'comment' not in chunks[lang] or chunks[lang]['comment'].find('</if>') == -1:
1846+ fdo.write(' </if>\n')
1847+ is_in_if = False
1848+ ours = ours[1:]
1849+ if 'comment' in chunks[lang]:
1850+ for s in chunks[lang]['comment'].split('\n')[:-1]:
1851+ if chunks[lang]['in_if'] is True and is_in_if and s.find('<if ') > -1:
1852+ continue
1853+ if s.find('<!-- No translations available. -->') > -1:
1854+ continue
1855+ fdo.write(s + '\n')
1856+ fdo.write(chunks[lang]['line'])
1857+ if ours[0] == tlang:
1858+ ours = ours[1:]
1859+ is_in_if = chunks[lang]['in_if']
1860+ if len(chunks.keys()) > 0:
1861+ while len(ours) > 0:
1862+ f = "%s_%s.%s" % (chunks[lang]['name'], ours[0], chunks[lang]['ext'])
1863+ if has_ifs and is_in_if is False:
1864+ fdo.write(' <if expr="pp_ifdef(\'use_third_party_translations\')">\n')
1865+ fdo.write(' %s<output filename="%s" type="data_package" lang="%s" />\n' % \
1866+ (' ' if has_ifs else '', f, ours[0]))
1867+ is_in_if = True
1868+ ours = ours[1:]
1869+ if has_ifs and is_in_if:
1870+ fdo.write(' </if>\n')
1871+ is_in_if = False
1872+ if c is not None:
1873+ for s in c.split('\n')[:-1]:
1874+ if s.find('<!-- No translations available. -->') > -1:
1875+ continue
1876+ if s.find('</if>') > -1:
1877+ continue
1878+ fdo.write(s + '\n')
1879+ if pak_found:
1880+ pak_saved.append(line)
1881+ continue
1882+ fdo.write(line)
1883+ fdi.close()
1884+ fdo.close()
1885+
1886+ def create_build_gyp_patch(self, directory, build_gyp_file, other_grd_files, nlangs,
1887+ whitelisted_new_langs = None):
1888+ # read the list of langs supported upstream
1889+ fd = open(build_gyp_file, "r")
1890+ data = fd.read()
1891+ fd.close()
1892+ r = data[data.find("'locales':"):]
1893+ olangs = sorted(re.findall("'(.*?)'", r[r.find('['):r.find(']')]))
1894+ # check for an optional use_third_party_translations list of locales
1895+ tpt = data.find('use_third_party_translations==1')
1896+ if tpt > 0:
1897+ tpt += data[tpt:].find("'locales':")
1898+ r = data[tpt:]
1899+ tptlangs = sorted(re.findall("'(.*?)'", r[r.find('['):r.find(']')]))
1900+ if nlangs == sorted(tptlangs + olangs):
1901+ return tptlangs
1902+ else:
1903+ if nlangs == olangs:
1904+ return []
1905+ # check if we need to only activate some whitelisted new langs
1906+ xlangs = None
1907+ nnlangs = [ x for x in nlangs if x not in olangs ]
1908+ if whitelisted_new_langs is not None:
1909+ if tpt > 0:
1910+ nlangs = [ x for x in nlangs if x not in olangs and x in whitelisted_new_langs ]
1911+ else:
1912+ xlangs = [ x for x in nlangs if x not in olangs and x not in whitelisted_new_langs ]
1913+ nlangs = [ x for x in nlangs if x in olangs or x in whitelisted_new_langs ]
1914+ elif tpt > 0:
1915+ nlangs = [ x for x in nlangs if x not in olangs ]
1916+
1917+ # we need a patch
1918+ if tpt > 0:
1919+ pos = tpt + data[tpt:].find('[')
1920+ end = data[:pos + 1]
1921+ ndata = end[:]
1922+ else:
1923+ pos = data.find("'locales':")
1924+ begin = data[pos:]
1925+ end = data[:pos + begin.find('\n')]
1926+ ndata = end[:]
1927+ end = data[pos + data[pos:].find(']'):]
1928+
1929+ # list of langs, folded
1930+ if len(nlangs) > 9:
1931+ ndata += '\n' + \
1932+ '\n'.join(textwrap.wrap("'" + "', '".join(nlangs) + "'",
1933+ break_long_words=False, width=76,
1934+ drop_whitespace=False,
1935+ expand_tabs=False,
1936+ replace_whitespace=False,
1937+ initial_indent=' ',
1938+ subsequent_indent=' ',
1939+ break_on_hyphens=False)) + '\n '
1940+ else:
1941+ ndata += "'%s'" % "', '".join(nlangs)
1942+
1943+ ndata += end
1944+
1945+ # write the patch
1946+ fromfile = "old/" + build_gyp_file
1947+ tofile = "new/" + build_gyp_file
1948+ fromdate = datetime.fromtimestamp(self.template.get_mtime(build_gyp_file)).strftime("%Y-%m-%d %H:%M:%S.%f000 +0000")
1949+ fromlines = [ x for x in re.split('(.*\n?)', data) if x != '' ]
1950+ todate = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f000 +0000")
1951+ tolines = [ x for x in re.split('(.*\n?)', ndata) if x != '' ]
1952+ patch = codecs.open(os.path.join(directory, "build.patch"), "wb", encoding="utf-8")
1953+ diff = unified_diff(fromlines, tolines, fromfile, tofile, fromdate, todate, n=3)
1954+ patch.write("diff -Nur %s %s\n" % (fromfile, tofile))
1955+ patch.writelines(''.join(diff))
1956+
1957+ for grd in other_grd_files:
1958+ grd_out = os.path.join(directory, os.path.basename(grd))
1959+ self.update_supported_langs_in_grd(grd, grd_out, langs)
1960+ if filecmp.cmp(grd, grd_out) == True:
1961+ os.unlink(grd_out)
1962+ continue # files are the same
1963+ # add it to the patch
1964+ fromfile = "old/" + grd
1965+ tofile = "new/" + grd
1966+ fromdate = datetime.fromtimestamp(self.template.get_mtime(grd)).strftime("%Y-%m-%d %H:%M:%S.%f000 +0000")
1967+ fromlines = codecs.open(grd, 'rb', encoding="utf-8").readlines()
1968+ todate = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f000 +0000")
1969+ tolines = codecs.open(grd_out, 'rb', encoding="utf-8").readlines()
1970+ diff = unified_diff(fromlines, tolines, fromfile, tofile, fromdate, todate, n=3)
1971+ patch.write("diff -Nur %s %s\n" % (fromfile, tofile))
1972+ patch.writelines(''.join(diff))
1973+ os.unlink(grd_out)
1974+ patch.close()
1975+ return nnlangs
1976+
1977+def usage():
1978+ print """
1979+Usage: %s [options] [grd_file [more_grd_files]]
1980+
1981+ Convert Chromium translation files (grd/xtb) into gettext files (pot/po) and back
1982+
1983+ options could be:
1984+ -d | --debug debug mode
1985+ -v | --verbose verbose mode
1986+ -h | --help this help screen
1987+
1988+ --export-gettext dir
1989+ export pot/po gettext files to dir
1990+
1991+ --import-gettext dir[,dir2][...]
1992+ import gettext pot/po files from those directories.
1993+ Directories must be ordered from the oldest to
1994+ the freshest. Only strings different from the grit
1995+ (upstream) translations are considered.
1996+
1997+ --import-grit-branch name:dir:grd1[,grd2,...]]
1998+ import the Grit files for this branch from this
1999+ directory. --import-grit-branch could be used several
2000+ times, and then, branches must be specified from the
2001+ freshest (trunk) to the more stable ones.
2002+ The default value is trunk:<cwd>
2003+ Note: must not be used along with --export-grit
2004+
2005+ --export-grit dir
2006+ export grd/xtb grit files to dir
2007+
2008+ --copy-grit dir copy the src grit files containing strings to dir
2009+ (useful to create diffs after --export-grit)
2010+
2011+ --whitelisted-new-langs lang1[,lang2][..]
2012+ comma separated list of new langs that have to be enabled
2013+ (assuming they have some strings translated). The default
2014+ is to enable all new langs, but for stable builds, a good
2015+ enough coverage is preferred
2016+
2017+ --create-patches dir
2018+ create unified patches per template in dir
2019+ (only useful after --export-grit)
2020+
2021+ --build-gyp-file file
2022+ location of the build/common.gypi file, used only
2023+ with --create-patches to add all new langs
2024+ for which we merged translated strings
2025+
2026+ --other-grd-files file1[,file2][..]
2027+ comma separated list of grd files to also patch
2028+ to add new langs for (see --build-gyp-file)
2029+
2030+ --html-output produce nice some HTML as output (on stdout)
2031+
2032+ --json-branches-info file
2033+ location of a json file containing the url, revision
2034+ and last date of both the upstream branch and
2035+ launchpad export. optionally used in the html output
2036+
2037+ --map-template-names new1=old1[,new2=old2][...]
2038+ comma separated list of template names mappings.
2039+ It is useful when upstream renames a grd file in a branch
2040+ to preserve the old name in gettext for the older branches
2041+
2042+ --landable-templates template1[,template2][...]
2043+ comma separated list of templates that are landable upstream
2044+ for all langs
2045+
2046+ --unlandable-templates template1[,template2][...]
2047+ comma separated list of templates that are not landable upstream,
2048+ even for new langs
2049+
2050+ --test-strcvt run the grit2gettext2grit checker
2051+ --test-conditions run the conditions evaluation checker
2052+
2053+""" % sys.argv[0].rpartition('/')[2]
2054+
2055+if '__main__' == __name__:
2056+ sys.stdout = codecs.getwriter('utf8')(sys.stdout)
2057+ try:
2058+ opts, args = getopt.getopt(sys.argv[1:], "dhv",
2059+ [ "test-strcvt", "test-conditions", "debug", "verbose", "help", "copy-grit=",
2060+ "import-grit-branch=", "export-gettext=", "import-gettext=", "export-grit=",
2061+ "create-patches=", "build-gyp-file=", "other-grd-files=",
2062+ "landable-templates=", "unlandable-templates=", "map-template-names=",
2063+ "whitelisted-new-langs=", "html-output", "json-branches-info=" ])
2064+ except getopt.GetoptError, err:
2065+ print str(err)
2066+ usage()
2067+ sys.exit(2)
2068+
2069+ verbose = False
2070+ debug = False
2071+ html_output = False
2072+ outdir = None
2073+ indir = None
2074+ export_gettext = None
2075+ import_gettext = None
2076+ export_grit = None
2077+ copy_grit = None
2078+ create_patches = None
2079+ build_gyp_file = None
2080+ json_info = None
2081+ other_grd_files = []
2082+ whitelisted_new_langs = None
2083+ templatenames_mapping = {}
2084+ landable_templates = []
2085+ unlandable_templates = []
2086+ branches = None
2087+ nbranches = []
2088+ for o, a in opts:
2089+ if o in ("-v", "--verbose"):
2090+ verbose = True
2091+ elif o in ("-h", "--help"):
2092+ usage()
2093+ sys.exit()
2094+ elif o in ("-d", "--debug"):
2095+ debug = True
2096+ elif o == "--import-grit-branch":
2097+ if branches is None:
2098+ branches = {}
2099+ branch, dir, grds = a.split(':')
2100+ for grd in grds.split(','):
2101+ name = os.path.basename(grd)
2102+ if name not in branches:
2103+ branches[name] = []
2104+ branches[name].append({ 'branch': branch, 'dir': dir, 'grd': grd })
2105+ if branch not in nbranches:
2106+ nbranches.append(branch)
2107+ elif o == "--export-gettext":
2108+ export_gettext = a
2109+ elif o == "--import-gettext":
2110+ import_gettext = a.split(",")
2111+ elif o == "--export-grit":
2112+ export_grit = a
2113+ elif o == "--copy-grit":
2114+ copy_grit = a
2115+ elif o == "--whitelisted-new-langs":
2116+ whitelisted_new_langs = a.split(",")
2117+ elif o == "--create-patches":
2118+ create_patches = a
2119+ elif o == "--build-gyp-file":
2120+ build_gyp_file = a
2121+ elif o == "--other-grd-files":
2122+ other_grd_files = a.split(',')
2123+ elif o == "--html-output":
2124+ html_output = True
2125+ elif o == "--json-branches-info":
2126+ json_info = a
2127+ elif o == "--landable-templates":
2128+ landable_templates = a.split(",")
2129+ elif o == "--unlandable-templates":
2130+ unlandable_templates = a.split(",")
2131+ elif o == "--map-template-names":
2132+ for c in a.split(','):
2133+ x = c.split('=')
2134+ templatenames_mapping[x[0]] = x[1]
2135+ elif o == "--test-strcvt":
2136+ StringCvt().test()
2137+ sys.exit()
2138+ elif o == "--test-conditions":
2139+ EvalConditions().test()
2140+ sys.exit()
2141+ else:
2142+ assert False, "unhandled option"
2143+
2144+ if branches is None and len(args) != 0:
2145+ branches = {}
2146+ for arg in args:
2147+ branches[os.path.basename(arg)] = [ { 'branch': 'default', 'dir': os.getcwd(), 'grd': arg } ]
2148+ if branches is None:
2149+ print "Please specify at least one grd file or use --import-grit-branch"
2150+ usage()
2151+ sys.exit(2)
2152+
2153+ # re-map the templates, if needed
2154+ for grd in templatenames_mapping.keys():
2155+ new = os.path.basename(grd)
2156+ old = os.path.basename(templatenames_mapping[grd])
2157+ if new in branches:
2158+ if old not in branches:
2159+ branches[old] = []
2160+ for branch in branches[new]:
2161+ branch['mapped_grd'] = old
2162+ branches[old].extend(branches[new])
2163+ # re-sort the branches
2164+ branches[old] = sorted(branches[old], lambda x,y: cmp(nbranches.index(x['branch']), nbranches.index(y['branch'])))
2165+ del(branches[new])
2166+
2167+ if html_output:
2168+ print """\
2169+<html>
2170+<head><meta charset="UTF-8">
2171+</head><body>
2172+<style type="text/css">
2173+body {
2174+ font-family: UbuntuBeta,Ubuntu,"Bitstream Vera Sans","DejaVu Sans",Tahoma,sans-serif;
2175+}
2176+div#legend {
2177+ float: left;
2178+}
2179+fieldset {
2180+ border-width: 1px;
2181+ border-color: #f0f0f0;
2182+}
2183+div#legend fieldset, div#branches fieldset {
2184+ border-width: 0px;
2185+}
2186+legend {
2187+ font-size: 80%;
2188+}
2189+div#branches {
2190+ float: left;
2191+ padding-left: 40px;
2192+}
2193+div#branches td {
2194+ padding-right: 5px;
2195+}
2196+div#stats {
2197+ padding-top: 5px;
2198+ clear: both;
2199+}
2200+a {
2201+ text-decoration: none;
2202+}
2203+a.l:link, a.l:visited {
2204+ color: black;
2205+}
2206+.error {
2207+ font-size: 90%;
2208+}
2209+div.error a {
2210+ font-family: monospace;
2211+ font-size: 120%;
2212+}
2213+table {
2214+ border-collapse: collapse;
2215+ border-spacing: 1px;
2216+ font-size: 0.9em;
2217+}
2218+th {
2219+ font-weight: bold;
2220+ color: #666;
2221+ padding-right: 5px;
2222+}
2223+th, td {
2224+ border: 1px #d2d2d2;
2225+ border-style: solid;
2226+ padding-left: 4px;
2227+ padding-top: 0px;
2228+ padding-bottom: 0px;
2229+}
2230+td.d {
2231+ font-size: 90%;
2232+ text-align: right;
2233+}
2234+td.n {
2235+ background: #FFA;
2236+}
2237+.lang {
2238+ font-weight: bold;
2239+ padding-left: 0.5em;
2240+ padding-right: 0.5em;
2241+ white-space: nowrap;
2242+}
2243+.progress_bar {
2244+ width: 100px; overflow: hidden; position: relative; padding: 0px;
2245+}
2246+.pb_label {
2247+ text-align: center; width: 100%;
2248+ position: absolute; z-index: 1001; left: 4px; top: -2px; color: white; font-size: 0.7em;
2249+}
2250+.pb_label2 {
2251+ text-align: center; width: 100%;
2252+ position: absolute; z-index: 1000; left: 5px; top: -1px; color: black; font-size: 0.7em;
2253+}
2254+.green_gradient {
2255+ height: 1em; position: relative; float: left;
2256+ background: #00ff00;
2257+ background: -moz-linear-gradient(top, #00ff00, #007700);
2258+ background: -webkit-gradient(linear, left top, left bottom, from(#00ff00), to(#007700));
2259+ filter: progid:DXImageTransform.Microsoft.Gradient(StartColorStr='#00ff00', EndColorStr='#007700', GradientType=0);
2260+}
2261+.red_gradient {
2262+ height: 1em; position: relative; float: left;
2263+ background: #ff8888;
2264+ background: -moz-linear-gradient(top, #ff8888, #771111);
2265+ background: -webkit-gradient(linear, left top, left bottom, from(#ff8888), to(#771111));
2266+ filter: progid:DXImageTransform.Microsoft.Gradient(StartColorStr='#ff8888', EndColorStr='#771111', GradientType=0);
2267+}
2268+.blue_gradient {
2269+ height: 1em; position: relative; float: left;
2270+ background: #62b0dd;
2271+ background: -moz-linear-gradient(top, #62b0dd, #1f3d4a);
2272+ background: -webkit-gradient(linear, left top, left bottom, from(#62b0dd), to(#1f3d4a));
2273+ filter: progid:DXImageTransform.Microsoft.Gradient(StartColorStr='#62b0dd', EndColorStr='#1f3d4a', GradientType=0);
2274+}
2275+.purple_gradient {
2276+ height: 1em; position: relative; float: left;
2277+ background: #b8a4ba;
2278+ background: -moz-linear-gradient(top, #b8a4ba, #5c3765);
2279+ background: -webkit-gradient(linear, left top, left bottom, from(#b8a4ba), to(#5c3765));
2280+ filter: progid:DXImageTransform.Microsoft.Gradient(StartColorStr='#b8a4ba', EndColorStr='#5c3765', GradientType=0);
2281+}
2282+</style>
2283+<script type="text/javascript" language="javascript">
2284+function progress_bar(where, red, green, purple, blue) {
2285+ var total = green + red + blue + purple;
2286+ if (total == 0)
2287+ total = 1;
2288+ var d = document.getElementById(where);
2289+ var v = 100 * (1 - (red / total));
2290+ if (total != 1) {
2291+ d.innerHTML += '<div class="pb_label">' + v.toFixed(1) + "%</div>";
2292+ d.innerHTML += '<div class="pb_label2">' + v.toFixed(1) + "%</div>";
2293+ }
2294+ else
2295+ d.style.width = "25px";
2296+ var pgreen = parseInt(100 * green / total);
2297+ var pblue = parseInt(100 * blue / total);
2298+ var ppurple = parseInt(100 * purple / total);
2299+ var pred = parseInt(100 * red / total);
2300+ if (pgreen + pblue + ppurple + pred != 100) {
2301+ if (red > 0)
2302+ pred = 100 - pgreen - pblue - ppurple;
2303+ else if (purple > 0)
2304+ ppurple = 100 - pgreen - pblue;
2305+ else if (blue > 0)
2306+ pblue = 100 - pgreen;
2307+ else
2308+ pgreen = 100;
2309+ }
2310+ if (green > 0)
2311+ d.innerHTML += '<div class="green_gradient" style="width:' + pgreen + '%;"></div>';
2312+ if (blue > 0)
2313+ d.innerHTML += '<div class="blue_gradient" style="width:' + pblue + '%;"></div>';
2314+ if (purple > 0)
2315+ d.innerHTML += '<div class="purple_gradient" style="width:' + ppurple + '%;"></div>';
2316+ if (red > 0)
2317+ d.innerHTML += '<div class="red_gradient" style="width:' + pred + '%;"></div>';
2318+ return true;
2319+}
2320+
2321+function toggle(e) {
2322+ var elt = document.getElementById(e + "-t");
2323+ var text = document.getElementById(e);
2324+ if (elt.style.display == "block") {
2325+ elt.style.display = "none";
2326+ text.innerHTML = "+";
2327+ }
2328+ else {
2329+ elt.style.display = "block";
2330+ text.innerHTML = "-";
2331+ }
2332+}
2333+
2334+function time_delta(date, e) {
2335+ var now = new Date();
2336+ var d = new Date(date);
2337+ var delta = (now - d) / 1000;
2338+ var elt = document.getElementById(e);
2339+ if (delta >= 3600) {
2340+ var h = parseInt(delta / 3600);
2341+ var m = parseInt((delta - h * 3600) / 60);
2342+ elt.innerHTML = '(' + h + 'h ' + m + 'min ago)';
2343+ return;
2344+ }
2345+ if (delta >= 60) {
2346+ var m = parseInt(delta / 60);
2347+ elt.innerHTML = '(' + m + 'min ago)';
2348+ return;
2349+ }
2350+ elt.innerHTML = '(seconds ago)';
2351+}
2352+
2353+</script>
2354+"""
2355+
2356+ prefix = os.path.commonprefix([ branches[x][0]['grd'] for x in branches.keys() ])
2357+ changes = 0
2358+ langs = []
2359+ mapped_langs = {}
2360+ cvts = {}
2361+ for grd in branches.keys():
2362+ cvts[grd] = Converter(branches[grd][0]['grd'],
2363+ lang_mapping = lang_mapping,
2364+ template_mapping = templatenames_mapping,
2365+ debug = debug,
2366+ html_output = html_output,
2367+ branches = branches[grd])
2368+
2369+ if cvts[grd].get_supported_strings_count() == 0:
2370+ if debug:
2371+ print "no string found in %s" % grd
2372+ if export_grit is not None and copy_grit is None:
2373+ directory = os.path.join(export_grit, os.path.dirname(branches[grd][0]['grd'])[len(prefix):])
2374+ if not os.path.isdir(directory):
2375+ os.makedirs(directory, 0755)
2376+ shutil.copy2(branches[grd][0]['grd'], directory)
2377+ continue
2378+
2379+ if copy_grit is not None:
2380+ cvts[grd].copy_grit_files(copy_grit)
2381+
2382+ if import_gettext is not None:
2383+ for directory in import_gettext:
2384+ cvts[grd].import_gettext_po_files(directory)
2385+ langs.extend(cvts[grd].translations.keys())
2386+
2387+ if export_gettext is not None:
2388+ cvts[grd].export_gettext_files(export_gettext)
2389+ changes += cvts[grd].template_changes + cvts[grd].translations_changes
2390+
2391+ # as we need to add all supported langs to the <outputs> section of all grd files,
2392+ # we have to wait for all the 'po' files to be imported and merged before we export
2393+ # the grit files and create the patches.
2394+
2395+ # supported langs
2396+ langs.append('en-US') # special case, it's not translated, but needs to be here
2397+ for lang in [ 'no' ]: # workaround for cases like the infamous no->nb mapping
2398+ while lang in langs:
2399+ langs.remove(lang)
2400+ langs.append(lang_mapping[lang])
2401+ r = {}
2402+ langs = sorted([ r.setdefault(e, e) for e in langs if e not in r ])
2403+
2404+ for grd in branches.keys():
2405+ if export_grit is not None:
2406+ cvts[grd].export_grit_files(os.path.join(export_grit, os.path.dirname(branches[grd][0]['grd'])[len(prefix):]), langs)
2407+ for lang in cvts[grd].template.mapped_langs:
2408+ mapped_langs[lang] = cvts[grd].template.mapped_langs[lang]['gettext']
2409+ if create_patches is not None:
2410+ cvts[grd].create_patches(create_patches)
2411+
2412+ # patch the build/common.gypi file if we have to
2413+ nlangs = None
2414+ if create_patches is not None and build_gyp_file is not None:
2415+ nlangs = cvts[branches.keys()[0]].create_build_gyp_patch(create_patches, build_gyp_file, other_grd_files, langs,
2416+ whitelisted_new_langs)
2417+
2418+ if create_patches is None:
2419+ # no need to display the stats
2420+ exit(1 if changes > 0 else 0)
2421+
2422+ # display some stats
2423+ html_js = ""
2424+ if html_output:
2425+ print """
2426+<p>
2427+<div>
2428+<div id="legend">
2429+<fieldset><legend>Legend</legend>
2430+<table border="0">
2431+<tr><td><div id='green_l' class='progress_bar'></td><td>translated upstream</td></tr>
2432+<tr><td><div id='blue_l' class='progress_bar'></td><td>translations updated in Launchpad</td></tr>
2433+<tr><td><div id='purple_l' class='progress_bar'></td><td>translated in Launchpad</td></tr>
2434+<tr><td><div id='red_l' class='progress_bar'></td><td>untranslated</td></tr>
2435+</table>
2436+</fieldset>
2437+</div>
2438+"""
2439+ html_js += "progress_bar('%s', %d, %d, %d, %d);\n" % ('green_l', 0, 1, 0, 0)
2440+ html_js += "progress_bar('%s', %d, %d, %d, %d);\n" % ('blue_l', 0, 0, 0, 1)
2441+ html_js += "progress_bar('%s', %d, %d, %d, %d);\n" % ('purple_l', 0, 0, 1, 0)
2442+ html_js += "progress_bar('%s', %d, %d, %d, %d);\n" % ('red_l', 1, 0, 0, 0)
2443+ if json_info:
2444+ now = datetime.utcfromtimestamp(os.path.getmtime(json_info)).strftime("%a %b %e %H:%M:%S UTC %Y")
2445+ binfo = json.loads(open(json_info, "r").read())
2446+ print """
2447+<div id="branches">
2448+<fieldset><legend>Last update info</legend>
2449+<table border="0">
2450+<tr><th>Branch</th><th>Revision</th><th>Date</th></tr>
2451+<tr><td><a href="%s">Upstream</a></td><td>r%s</td><td>%s <em id='em-u'></em> </td></tr>
2452+<tr><td><a href="%s">Launchpad export</a></td><td>r%s</td><td>%s <em id='em-lp'></em> </td></tr>
2453+<tr><td>This page</a></td><td>-</td><td>%s <em id='em-now'></em> </td></tr>
2454+</table>
2455+</fieldset>
2456+</div>
2457+""" % (binfo['upstream']['url'], binfo['upstream']['revision'], binfo['upstream']['date'],
2458+ binfo['launchpad-export']['url'], binfo['launchpad-export']['revision'],
2459+ binfo['launchpad-export']['date'], now)
2460+ html_js += "time_delta('%s', '%s');\n" % (binfo['upstream']['date'], 'em-u')
2461+ html_js += "time_delta('%s', '%s');\n" % (binfo['launchpad-export']['date'], 'em-lp')
2462+ html_js += "time_delta('%s', '%s');\n" % (now, 'em-now')
2463+ print """
2464+<div id="stats">
2465+<table border="0">
2466+<tr><th rowspan="2">Rank</th><th rowspan="2">Lang</th><th colspan='5'>TOTAL</th><th colspan='5'>"""
2467+ print ("</th><th colspan='5'>".join([ "%s (<a href='http://git.chromium.org/gitweb/?p=chromium.git;a=history;f=%s;hb=HEAD'>+</a>)" \
2468+ % (os.path.splitext(grd)[0], branches[grd][0]['grd']) \
2469+ for grd in sorted(branches.keys()) ])) + "</th></tr><tr>"
2470+ j = 0
2471+ for grd in [ 'TOTAL' ] + sorted(branches.keys()):
2472+ print """
2473+<th>Status</th>
2474+<th><div id='%s_t%d' class='progress_bar'></th>
2475+<th><div id='%s_t%d' class='progress_bar'></th>
2476+<th><div id='%s_t%d' class='progress_bar'></th>
2477+<th><div id='%s_t%d' class='progress_bar'></th>""" % ('red', j, 'green', j, 'purple', j, 'blue', j)
2478+ html_js += "progress_bar('%s_t%d', %d, %d, %d, %d);\n" % ('green', j, 0, 1, 0, 0)
2479+ html_js += "progress_bar('%s_t%d', %d, %d, %d, %d);\n" % ('blue', j, 0, 0, 0, 1)
2480+ html_js += "progress_bar('%s_t%d', %d, %d, %d, %d);\n" % ('purple', j, 0, 0, 1, 0)
2481+ html_js += "progress_bar('%s_t%d', %d, %d, %d, %d);\n" % ('red', j, 1, 0, 0, 0)
2482+ j += 1
2483+ print "</tr>"
2484+ else:
2485+ print """\
2486+ +----------------------- % translated
2487+ | +----------------- untranslated
2488+ | | +------------ translated upstream
2489+ | | | +------- translated in Launchpad
2490+ | | | | +-- translations updated in Launchpad
2491+ | | | | |
2492+ V V V V V"""
2493+ print "-- lang -- " + \
2494+ ' '.join([ (" %s " % os.path.splitext(grd)[0]).center(25, "-") \
2495+ for grd in [ 'TOTAL' ] + sorted(branches.keys()) ])
2496+ totals = {}
2497+ for lang in langs:
2498+ klang = lang
2499+ if lang == 'nb':
2500+ klang = 'no'
2501+ totals[klang] = { 'total': 0, 'missing': 0, 'translated_upstream': 0, 'new': 0, 'updated': 0, 'lskipped': 0 }
2502+ for grd in branches.keys():
2503+ tot, lskipped = cvts[grd].template.get_supported_strings_count(klang)
2504+ totals[klang]['lskipped'] += lskipped
2505+ totals[klang]['total'] += tot
2506+ totals[klang]['missing'] += tot
2507+ if klang in cvts[grd].template.stats:
2508+ totals[klang]['missing'] -= cvts[grd].template.stats[klang]['translated_upstream'] + \
2509+ cvts[grd].template.stats[klang]['new'] + cvts[grd].template.stats[klang]['updated']
2510+ totals[klang]['translated_upstream'] += cvts[grd].template.stats[klang]['translated_upstream']
2511+ totals[klang]['new'] += cvts[grd].template.stats[klang]['new']
2512+ totals[klang]['updated'] += cvts[grd].template.stats[klang]['updated']
2513+
2514+ rank = 0
2515+ p_rank = 0
2516+ p_score = -1
2517+ t_landable = 0
2518+ for lang in sorted(totals, lambda x, y: cmp("%05d %05d %s" % (totals[x]['missing'], totals[x]['total'] - totals[x]['updated'] - totals[x]['new'], x),
2519+ "%05d %05d %s" % (totals[y]['missing'], totals[y]['total'] - totals[y]['updated'] - totals[y]['new'], y))):
2520+ if lang == 'en-US':
2521+ continue
2522+ rank += 1
2523+ if p_score != totals[lang]['missing']:
2524+ p_score = totals[lang]['missing']
2525+ p_rank = rank
2526+ rlang = lang
2527+ if lang in lang_mapping:
2528+ rlang = lang_mapping[lang]
2529+ if html_output:
2530+ s = "<tr><td>%s</td><td class='lang'><a class='l' href='%s'>%s</a></td>" % \
2531+ ("#%d" % p_rank, 'https://translations.launchpad.net/chromium-browser/translations/+lang/' + \
2532+ mapped_langs[lang], rlang)
2533+ s += "<td><div id='%s' class='progress_bar'></div></td>" % rlang
2534+ s += "<td class='d'>%d</td><td class='d'>%d</td><td class='d'>%d</td><td class='d'>%d</td>" % \
2535+ (totals[lang]['missing'], totals[lang]['translated_upstream'],
2536+ totals[lang]['new'], totals[lang]['updated'])
2537+ html_js += "progress_bar('%s', %d, %d, %d, %d);\n" % \
2538+ (rlang, totals[lang]['missing'], totals[lang]['translated_upstream'],
2539+ totals[lang]['new'], totals[lang]['updated'])
2540+ else:
2541+ s = "%-3s %-6s " % ("#%d" % p_rank, rlang)
2542+ s += "%3d%% %4d %4d %4d %4d" % \
2543+ (100.0 * float(totals[lang]['total'] - totals[lang]['missing']) / float(totals[lang]['total']),
2544+ totals[lang]['missing'], totals[lang]['translated_upstream'],
2545+ totals[lang]['new'], totals[lang]['updated'])
2546+ j = 0
2547+ for grd in sorted(branches.keys()):
2548+ j += 1
2549+ tplt = os.path.splitext(grd)[0].replace('_', '-')
2550+ total, lskipped = cvts[grd].template.get_supported_strings_count(lang)
2551+ if lang in cvts[grd].template.stats:
2552+ missing = total - cvts[grd].template.stats[lang]['translated_upstream'] - \
2553+ cvts[grd].template.stats[lang]['new'] - cvts[grd].template.stats[lang]['updated']
2554+ if html_output:
2555+ if len(unlandable_templates) == 0 and len(landable_templates) == 0:
2556+ landable = False
2557+ else:
2558+ landable = (nlangs is not None and lang in nlangs and tplt not in unlandable_templates) or \
2559+ (nlangs is not None and lang not in nlangs and tplt in landable_templates)
2560+ if landable:
2561+ t_landable += cvts[grd].template.stats[lang]['new'] + cvts[grd].template.stats[lang]['updated']
2562+ s += "<td><div id='%s_%d' class='progress_bar'></div></td>" % (rlang, j)
2563+ s += "<td class='d'>%d</td><td class='d'>%d</td><td class='d%s'>%d</td><td class='d%s'>%d</td>" % \
2564+ (missing,
2565+ cvts[grd].template.stats[lang]['translated_upstream'],
2566+ " n" if landable and cvts[grd].template.stats[lang]['new'] > 0 else "",
2567+ cvts[grd].template.stats[lang]['new'],
2568+ " n" if landable and cvts[grd].template.stats[lang]['updated'] > 0 else "",
2569+ cvts[grd].template.stats[lang]['updated'])
2570+ html_js += "progress_bar('%s_%d', %d, %d, %d, %d);\n" % \
2571+ (rlang, j, missing,
2572+ cvts[grd].template.stats[lang]['translated_upstream'],
2573+ cvts[grd].template.stats[lang]['new'],
2574+ cvts[grd].template.stats[lang]['updated'])
2575+ else:
2576+ if float(total) > 0:
2577+ pct = 100.0 * float(total - missing) / float(total)
2578+ else:
2579+ pct = 0
2580+ s += " %3d%% %4d %4d %4d %4d" % \
2581+ (pct, missing,
2582+ cvts[grd].template.stats[lang]['translated_upstream'],
2583+ cvts[grd].template.stats[lang]['new'],
2584+ cvts[grd].template.stats[lang]['updated'])
2585+ else:
2586+ if html_output:
2587+ s += "<td><div id='%s_%d' class='progress_bar'></div></td>" % (rlang, j)
2588+ s += "<td class='d'>%d</td><td class='d'>%d</td><td class='d'>%d</td><td class='d'>%d</td>" % \
2589+ (total, 0, 0, 0)
2590+ html_js += "progress_bar('%s_%d', %d, %d, %d, %d);\n" % \
2591+ (rlang, j, total, 0, 0, 0)
2592+ else:
2593+ s += " %3d%% %4d %4d %4d %4d" % (0, total, 0, 0, 0)
2594+ if html_output:
2595+ s += "</tr>"
2596+ print s
2597+ if html_output:
2598+ landable_sum = ""
2599+ if t_landable > 0:
2600+ landable_sum = """<p>
2601+<div name='landable'>
2602+<table border="0"><tr><td class="d n">%d strings are landable upstream</td></tr></table></div>
2603+""" % t_landable
2604+ print """\
2605+</table>
2606+%s</div>
2607+</div>
2608+<script type="text/javascript" language="javascript">
2609+%s
2610+</script>
2611+</body>
2612+</html>""" % (landable_sum, html_js)
2613+ exit(1 if changes > 0 else 0)
2614+
2615
2616=== added file 'create-patches.sh'
2617--- create-patches.sh 1970-01-01 00:00:00 +0000
2618+++ create-patches.sh 2024-02-28 12:58:47 +0000
2619@@ -0,0 +1,185 @@
2620+#!/bin/sh
2621+
2622+# Create the translation patches (grit format) based on the last
2623+# translation export (gettext format)
2624+# (c) 2010-2011, Fabien Tassin <fta@ubuntu.com>
2625+
2626+# location of chromium2pot.py (lp:~chromium-team/chromium-browser/chromium-translations-tools.head)
2627+# (must already exist)
2628+BIN_DIR=/data/bot/chromium-translations-tools.head
2629+
2630+# Launchpad translation export (must already exist, will be pulled here)
2631+LPE_DIR=/data/bot/upstream/chromium-translations-exports.head
2632+
2633+# local svn branches (updated by drobotik)
2634+SRC_TRUNK_DIR=/data/bot/upstream/chromium-browser.svn/src
2635+SRC_DEV_DIR=/data/bot/upstream/chromium-dev.svn/src
2636+SRC_BETA_DIR=/data/bot/upstream/chromium-beta.svn/src
2637+SRC_STABLE_DIR=/data/bot/upstream/chromium-stable.svn/src
2638+
2639+####
2640+
2641+OUT_DIR=$(mktemp -d)
2642+
2643+# List of options per branch
2644+NEW_OPTS="--map-template-names ui/base/strings/ui_strings.grd=ui/base/strings/app_strings.grd"
2645+OLD_OPTS=""
2646+
2647+OPTS_TRUNK=$NEW_OPTS
2648+OPTS_DEV=$NEW_OPTS
2649+OPTS_BETA=$NEW_OPTS
2650+OPTS_STABLE=$OLD_OPTS
2651+
2652+# List of templates to send to Launchpad
2653+NEW_TEMPLATES=$(cat - <<EOF
2654+ui/base/strings/ui_strings.grd
2655+EOF
2656+)
2657+OLD_TEMPLATES=$(cat - <<EOF
2658+ui/base/strings/app_strings.grd
2659+EOF
2660+)
2661+TEMPLATES_TRUNK=$NEW_TEMPLATES
2662+TEMPLATES_DEV=$NEW_TEMPLATES
2663+TEMPLATES_BETA=$NEW_TEMPLATES
2664+TEMPLATES_STABLE=$OLD_TEMPLATES
2665+
2666+TEMPLATES=$(cat - <<EOF
2667+chrome/app/chromium_strings.grd
2668+chrome/app/generated_resources.grd
2669+chrome/app/policy/policy_templates.grd
2670+webkit/glue/inspector_strings.grd
2671+webkit/glue/webkit_strings.grd
2672+EOF
2673+)
2674+
2675+# List of other templates to update for new langs, but that are
2676+# not sent to Launchpad
2677+NEW_OTHER_TEMPLATES=$(cat - <<EOF
2678+ui/base/strings/app_locale_settings.grd
2679+EOF
2680+)
2681+OLD_OTHER_TEMPLATES=$(cat - <<EOF
2682+app/resources/app_locale_settings.grd
2683+EOF
2684+)
2685+OTHER_TEMPLATES_TRUNK=$NEW_OTHER_TEMPLATES
2686+OTHER_TEMPLATES_DEV=$NEW_OTHER_TEMPLATES
2687+OTHER_TEMPLATES_BETA=$NEW_OTHER_TEMPLATES
2688+OTHER_TEMPLATES_STABLE=$NEW_OTHER_TEMPLATES
2689+# Common to all branches
2690+OTHER_TEMPLATES=$(cat - <<EOF
2691+chrome/app/resources/locale_settings.grd
2692+chrome/app/resources/locale_settings_linux.grd
2693+chrome/app/resources/locale_settings_cros.grd
2694+EOF
2695+)
2696+
2697+######
2698+
2699+TEMPLATES=$(echo $TEMPLATES | tr '[ \n]' ' ')
2700+
2701+(cd $LPE_DIR ; bzr pull -q)
2702+
2703+space_list () {
2704+ local V1="$1"
2705+ local V2="$2"
2706+ echo "$V1 $V2" | tr '[ \n]' ' ' | sed -e 's/ $//'
2707+}
2708+
2709+comma_list () {
2710+ local V1="$1"
2711+ local V2="$2"
2712+
2713+ echo "$V1 $V2" | tr '[ \n]' ',' | sed -e 's/,$//'
2714+}
2715+
2716+get_branches_info () {
2717+ local BRANCH=$1
2718+ local SRC_DIR=$2
2719+ local JSON=$3
2720+
2721+ # upstream url, revision & last change date
2722+ UURL=$(cd $SRC_DIR; svn info | grep '^URL: ' | sed -e 's/.*: //')
2723+ UREV="$(cd $SRC_DIR; svn info | grep '^Last Changed Rev:' | sed -e 's/.*: //') ($(cut -d= -f2 $SRC_DIR/chrome/VERSION | sed -e 's,$,.,' | tr -d '\n' | sed -e 's/.$//'))"
2724+ UDATE=$(date -d "$(cd $SRC_DIR; svn info | grep '^Last Changed Date:' | sed -e 's/.*: //')" --utc)
2725+
2726+ # Launchpad url, revision & last change date
2727+ LURL=$(cd $LPE_DIR; bzr info | grep 'parent branch' | sed -e 's,.*: bzr+ssh://bazaar,https://code,')
2728+ LREV=$(cd $LPE_DIR; bzr revno)
2729+ LDATE=$(date -d "$(cd $LPE_DIR; bzr info -v | grep 'latest revision' | sed -e 's/.*: //')" --utc)
2730+
2731+ cat - <<EOF > $JSON
2732+{
2733+ "upstream": {
2734+ "revision": "$UREV",
2735+ "url": "$UURL",
2736+ "date": "$UDATE"
2737+ },
2738+ "launchpad-export": {
2739+ "revision": $LREV,
2740+ "url": "$LURL",
2741+ "date": "$LDATE"
2742+ }
2743+}
2744+EOF
2745+}
2746+
2747+create_patch () {
2748+ local BRANCH=$1
2749+ local SRC_DIR=$2
2750+ local BRANCH_OPTS="$3"
2751+ local TEMPLATES="$4"
2752+ local OTHER_TEMPLATES="$5"
2753+
2754+ LOG=converter-output.html
2755+ DLOG=converter-diffstat.html
2756+
2757+ set -e
2758+ cd $SRC_DIR
2759+ mkdir -p $OUT_DIR/$BRANCH/new
2760+
2761+ get_branches_info $BRANCH $SRC_DIR $OUT_DIR/$BRANCH/new/revisions.json
2762+
2763+ local OPTS=""
2764+ if [ $BRANCH = trunk ] ; then
2765+ OPTS="--landable-templates chromium-strings,inspector-strings --unlandable-templates policy-templates"
2766+ fi
2767+
2768+ # Generate the new files, using the new template and the translations exported by launchpad
2769+ $BIN_DIR/chromium2pot.py \
2770+ --html-output \
2771+ --json-branches-info $OUT_DIR/$BRANCH/new/revisions.json \
2772+ --create-patches $OUT_DIR/$BRANCH/new/patches \
2773+ --import-gettext $LPE_DIR \
2774+ --export-grit $OUT_DIR/$BRANCH/new/patched-files \
2775+ --build-gyp-file build/common.gypi \
2776+ --other-grd-files $OTHER_TEMPLATES \
2777+ $OPTS \
2778+ $BRANCH_OPTS \
2779+ $TEMPLATES >> $OUT_DIR/$BRANCH/new/$LOG 2>&1
2780+ echo >> $OUT_DIR/$BRANCH/new/$LOG
2781+
2782+ ( cd "$OUT_DIR/$BRANCH/new/patches" ; for i in * ; do mv $i $i.txt ; done )
2783+ echo "<pre>" > $OUT_DIR/$BRANCH/new/$DLOG
2784+ ( cd "$OUT_DIR/$BRANCH/new" ; find patches -type f | xargs --verbose -n 1 diffstat -p 1 >> $DLOG 2>&1 )
2785+ perl -i -pe 's,^(diffstat -p 1 )(\S+)(.*),$1<a href="$2">$2</a>$3,;' $OUT_DIR/$BRANCH/new/$DLOG
2786+ echo "</pre>" >> $OUT_DIR/$BRANCH/new/$DLOG
2787+
2788+ # get the old files
2789+ mkdir $OUT_DIR/$BRANCH/old
2790+ lftp -e "lcd $OUT_DIR/$BRANCH/old; cd public_html/chromium/translations/$BRANCH; mirror; quit" sftp://people.ubuntu.com > /dev/null 2>&1
2791+ set +e
2792+ (cd $OUT_DIR/$BRANCH ; diff -Nur old new > diff.patch; cd old ; patch -p 1 < ../diff.patch > /dev/null 2>&1 )
2793+ set -e
2794+
2795+ lftp -e "lcd $OUT_DIR/$BRANCH/old; cd public_html/chromium/translations/$BRANCH; mirror --delete -R; quit" sftp://people.ubuntu.com > /dev/null 2>&1
2796+ set +e
2797+}
2798+
2799+create_patch "trunk" $SRC_TRUNK_DIR "$OPTS_TRUNK" "$(space_list "$TEMPLATES" "$TEMPLATES_TRUNK")" $(comma_list "$OTHER_TEMPLATES_TRUNK" "$OTHER_TEMPLATES")
2800+create_patch "dev" $SRC_DEV_DIR "$OPTS_DEV" "$(space_list "$TEMPLATES" "$TEMPLATES_DEV")" $(comma_list "$OTHER_TEMPLATES_DEV" "$OTHER_TEMPLATES")
2801+create_patch "beta" $SRC_BETA_DIR "$OPTS_BETA" "$(space_list "$TEMPLATES" "$TEMPLATES_BETA")" $(comma_list "$OTHER_TEMPLATES_BETA" "$OTHER_TEMPLATES")
2802+create_patch "stable" $SRC_STABLE_DIR "$OPTS_STABLE" "$(space_list "$TEMPLATES" "$TEMPLATES_STABLE")" $(comma_list "$OTHER_TEMPLATES_STABLE" "$OTHER_TEMPLATES")
2803+
2804+rm -rf $OUT_DIR
2805
2806=== added file 'desktop2gettext.py'
2807--- desktop2gettext.py 1970-01-01 00:00:00 +0000
2808+++ desktop2gettext.py 2024-02-28 12:58:47 +0000
2809@@ -0,0 +1,378 @@
2810+#!/usr/bin/python
2811+# -*- coding: utf-8 -*-
2812+
2813+# (c) 2010-2011, Fabien Tassin <fta@ubuntu.com>
2814+
2815+# Convert a desktop file to Gettext files and back
2816+
2817+import sys, getopt, os, codecs, re, time
2818+from datetime import datetime
2819+
2820+class DesktopFile(dict):
2821+ """ Read and write a desktop file """
2822+ def __init__(self, desktop = None, src_pkg = None, verbose = False):
2823+ self.changed = False
2824+ self.data = []
2825+ self.headers = {}
2826+ self.template = {}
2827+ self.translations = {}
2828+ self.src_pkg = src_pkg
2829+ self.mtime = None
2830+ self.verbose = verbose
2831+ if desktop is not None:
2832+ self.read_desktop_file(desktop)
2833+
2834+ def read_desktop_file(self, filename):
2835+ self.data = []
2836+ self.template = {}
2837+ self.mtime = os.path.getmtime(filename)
2838+ fd = codecs.open(filename, "rb", encoding="utf-8")
2839+ section = None
2840+ for line in fd.readlines():
2841+ m = re.match(r'^\[(.*?)\]', line)
2842+ if m is not None:
2843+ section = m.group(1)
2844+ assert section not in self.template, "Duplicate section [%s]" % section
2845+ self.template[section] = {}
2846+ m = re.match(r'^(Name|GenericName|Comment)(\[\S+\]|)=(.*)', line)
2847+ if m is None:
2848+ self.data.append(line)
2849+ continue
2850+ assert section is not None, "Found a '%s' outside a section" % m.group(1)
2851+ entry = m.group(1)
2852+ value = m.group(3)
2853+ if m.group(2) == "":
2854+ # master string
2855+ self.data.append(line)
2856+ assert entry not in self.template[section], \
2857+ "Duplicate entry '%s' in section [%s]" % (entry, section)
2858+ self.template[section][entry] = value
2859+ if value not in self.translations:
2860+ self.translations[value] = {}
2861+ else:
2862+ # translation
2863+ lang = m.group(2)[1:-1]
2864+ string = self.template[section][entry]
2865+ assert entry in self.template[section], \
2866+ "Translation found for lang '%s' in section [%s] before master entry '%s'" % \
2867+ (lang, section, entry)
2868+ if lang not in self.translations[string]:
2869+ self.translations[string][lang] = value
2870+ fd.close()
2871+
2872+ def dump(self):
2873+ for section in sorted(self.template.keys()):
2874+ print "[%s]:" % section
2875+ for entry in sorted(self.template[section].keys()):
2876+ print " '%s': '%s'" % (entry, self.template[section][entry])
2877+ print
2878+ for string in sorted(self.translations.keys()):
2879+ print "'%s':" % string
2880+ for lang in sorted(self.translations[string].keys()):
2881+ print " '%s' => '%s'" % (lang, self.translations[string][lang])
2882+
2883+ def write_desktop(self, file):
2884+ fd = codecs.open(file, "wb", encoding="utf-8")
2885+ for ent in self.data:
2886+ fd.write(ent)
2887+ m = re.match(r'^(Name|GenericName|Comment)=(.*)', ent)
2888+ if m is None:
2889+ continue
2890+ k = m.group(2)
2891+ if k not in self.translations:
2892+ continue
2893+ for lang in sorted(self.translations[k].keys()):
2894+ fd.write("%s[%s]=%s\n" % (m.group(1), lang, self.translations[k][lang]))
2895+
2896+ def write_gettext_header(self, fd, mtime = None, last_translator = None, lang_team = None):
2897+ mtime = "YEAR-MO-DA HO:MI+ZONE" if mtime is None else \
2898+ datetime.fromtimestamp(mtime).strftime("%Y-%m-%d %H:%M+0000")
2899+ last_translator = "FULL NAME <EMAIL@ADDRESS>" if last_translator is None else \
2900+ last_translator
2901+ lang_team = "LANGUAGE <LL@li.org>" if lang_team is None else lang_team
2902+ fd.write("""\
2903+# %s desktop file.
2904+# Copyright (C) 2010-2011 Fabien Tassin
2905+# This file is distributed under the same license as the %s package.
2906+# Fabien Tassin <fta@ubuntu.com>, 2010-2011.
2907+#
2908+msgid ""
2909+msgstr ""
2910+"Project-Id-Version: %s\\n"
2911+"Report-Msgid-Bugs-To: https://bugs.launchpad.net/ubuntu/+source/%s/+filebug\\n"
2912+"POT-Creation-Date: %s\\n"
2913+"PO-Revision-Date: %s\\n"
2914+"Last-Translator: %s\\n"
2915+"Language-Team: %s\\n"
2916+"MIME-Version: 1.0\\n"
2917+"Content-Type: text/plain; charset=UTF-8\\n"
2918+"Content-Transfer-Encoding: 8bit\\n"
2919+
2920+""" % (self.src_pkg, self.src_pkg, self.src_pkg, self.src_pkg,
2921+ datetime.fromtimestamp(self.mtime).strftime("%Y-%m-%d %H:%M+0000"),
2922+ mtime, last_translator, lang_team))
2923+
2924+ def write_gettext_string(self, fd, string, translation = None, usage = None):
2925+ if translation is None:
2926+ translation = ""
2927+ if usage is not None:
2928+ fd.write("#. %s\n" % usage)
2929+ fd.write('msgid "%s"\nmsgstr "%s"\n\n' % (string, translation))
2930+
2931+ def get_usage(self, string):
2932+ res = []
2933+ for section in sorted(self.template.keys()):
2934+ for entry in sorted(self.template[section].keys()):
2935+ if self.template[section][entry] == string:
2936+ res.append("[%s] %s" % (section, entry))
2937+ return ", ".join(res)
2938+
2939+ def read_gettext_string(self, fd):
2940+ string = {}
2941+ cur = None
2942+ while 1:
2943+ s = fd.readline()
2944+ if len(s) == 0 or s == "\n":
2945+ break # EOF or end of block
2946+ if s.rfind('\n') == len(s) - 1:
2947+ s = s[:-1] # chomp
2948+ if s.find("# ") == 0 or s == "#": # translator-comment
2949+ if 'comment' not in string:
2950+ string['comment'] = ''
2951+ string['comment'] += s[2:]
2952+ continue
2953+ if s.find("#:") == 0: # reference
2954+ if 'reference' not in string:
2955+ string['reference'] = ''
2956+ string['reference'] += s[2:]
2957+ if s[2:].find(" id: ") == 0:
2958+ string['id'] = s[7:].split(' ')[0]
2959+ continue
2960+ if s.find("#.") == 0: # extracted-comments
2961+ if 'extracted' not in string:
2962+ string['extracted'] = ''
2963+ string['extracted'] += s[2:]
2964+ if s[2:].find(" - condition: ") == 0:
2965+ if 'conditions' not in string:
2966+ string['conditions'] = []
2967+ string['conditions'].append(s[16:])
2968+ continue
2969+ if s.find("#~") == 0: # obsolete messages
2970+ continue
2971+ if s.find("#") == 0: # something else
2972+ print "%s not expected. Skip" % repr(s)
2973+ continue # not supported/expected
2974+ if s.find("msgid ") == 0:
2975+ cur = "string"
2976+ if cur not in string:
2977+ string[cur] = u""
2978+ else:
2979+ string[cur] += "\n"
2980+ string[cur] += s[6:]
2981+ continue
2982+ if s.find("msgstr ") == 0:
2983+ cur = "translation"
2984+ if cur not in string:
2985+ string[cur] = u""
2986+ else:
2987+ string[cur] += "\n"
2988+ string[cur] += s[7:]
2989+ continue
2990+ if s.find('"') == 0:
2991+ if cur is None:
2992+ print "'%s' not expected here. Skip" % s
2993+ continue
2994+ string[cur] += "\n" + s
2995+ continue
2996+ print "'%s' not expected here. Skip" % s
2997+ return None if string == {} else string
2998+
2999+ def merge_gettext_string(self, string, lang):
3000+ msg = string['string'][1:-1]
3001+ if msg == "": # header
3002+ self.headers[lang] = {}
3003+ map(lambda x: self.headers[lang].setdefault(x.split(": ")[0], x.split(": ")[1]),
3004+ string['translation'][4:-3].replace('\\n"\n"', '\n').replace('"\n"', '').split('\n'))
3005+ return
3006+ if msg not in self.translations:
3007+ return # obsolete string
3008+ translation = string['translation'][1:-1]
3009+ if translation == "": # no translation
3010+ return
3011+ if lang not in self.translations[msg]:
3012+ if self.verbose:
3013+ print "merge translation for lang '%s' string '%s'" % (lang, msg)
3014+ self.translations[msg][lang] = translation
3015+ elif self.translations[msg][lang] != translation:
3016+ if self.verbose:
3017+ print "update translation for lang '%s' string '%s'" % (lang, msg)
3018+ self.translations[msg][lang] = translation
3019+
3020+ def read_gettext_file(self, filename):
3021+ fd = codecs.open(filename, "rb", encoding="utf-8")
3022+ strings = []
3023+ while 1:
3024+ string = self.read_gettext_string(fd)
3025+ if string is None:
3026+ break
3027+ strings.append(string)
3028+ fd.close()
3029+ return strings
3030+
3031+ def get_langs(self):
3032+ langs = []
3033+ for st in self.translations:
3034+ for lang in self.translations[st]:
3035+ if lang not in langs:
3036+ langs.append(lang)
3037+ return sorted(langs)
3038+
3039+ def export_gettext_file(self, directory, template_name = "desktop_file", lang = None):
3040+ filename = os.path.join(directory, "%s.pot" % template_name if lang is None else "%s.po" % lang)
3041+ # if there's already a file with this name, compare its content and only update it
3042+ # when it's needed
3043+ update = False
3044+ if not os.path.exists(filename):
3045+ update = True
3046+ else:
3047+ # compare the strings
3048+ strings = self.read_gettext_file(filename)[1:]
3049+ old = sorted(map(lambda x: x['string'][1:-1], strings))
3050+ new = sorted(self.translations.keys())
3051+ if old != new:
3052+ update = True
3053+ if self.verbose:
3054+ print "strings differ for %s. Update" % filename
3055+ if not update:
3056+ # compare string descriptions
3057+ old = map(lambda x: x['extracted'][1:],
3058+ sorted(strings, lambda a, b: cmp(a['string'][1:-1], b['string'][1:-1])))
3059+ new = map(lambda x: self.get_usage(x), sorted(self.translations.keys()))
3060+ if old != new:
3061+ update = True
3062+ if self.verbose:
3063+ print "string descriptions differ for %s. Update" % filename
3064+ if not update and lang is not None:
3065+ # compare translations
3066+ old = map(lambda x: x['translation'][1:-1],
3067+ sorted(strings, lambda a, b: cmp(a['string'][1:-1], b['string'][1:-1])))
3068+ new = map(lambda x: self.translations[x][lang] if lang in self.translations[x] else "",
3069+ sorted(self.translations.keys()))
3070+ if old != new:
3071+ update = True
3072+ if self.verbose:
3073+ print "translations differ for %s. Update" % filename
3074+ if not update:
3075+ return
3076+ if self.verbose:
3077+ print "update %s" % filename
3078+ fd = codecs.open(filename, 'wb', encoding='utf-8')
3079+ last_translator = None
3080+ lang_team = None
3081+ if lang is None:
3082+ self.mtime = time.time()
3083+ mtime = None
3084+ else:
3085+ mtime = time.time()
3086+ if lang in self.headers and 'Last-Translator' in self.headers[lang]:
3087+ last_translator = self.headers[lang]['Last-Translator']
3088+ if lang in self.headers and 'Language-Team' in self.headers[lang]:
3089+ lang_team = self.headers[lang]['Language-Team']
3090+ else:
3091+ lang_team = "%s <%s@li.org>" % (lang, lang)
3092+ self.write_gettext_header(fd, mtime = mtime, last_translator = last_translator, lang_team = lang_team)
3093+ for string in sorted(self.translations.keys()):
3094+ val = self.translations[string][lang] \
3095+ if lang is not None and lang in self.translations[string] else None
3096+ self.write_gettext_string(fd, string, val, usage = self.get_usage(string))
3097+ fd.close()
3098+ self.changed = True
3099+
3100+ def export_gettext_files(self, directory):
3101+ if not os.path.isdir(directory):
3102+ os.makedirs(directory, 0755)
3103+ self.export_gettext_file(directory)
3104+ for lang in self.get_langs():
3105+ self.export_gettext_file(directory, lang = lang)
3106+
3107+ def import_gettext_files(self, directory):
3108+ """ Import strings from gettext 'po' files, ignore the 'pot'.
3109+ Only merge strings matching our desktop file """
3110+ assert os.path.isdir(directory)
3111+ for file in os.listdir(directory):
3112+ lang, ext = os.path.splitext(file)
3113+ if ext != '.po':
3114+ continue
3115+ for string in self.read_gettext_file(os.path.join(directory, file)):
3116+ self.merge_gettext_string(string, lang)
3117+
3118+def usage():
3119+ appname = sys.argv[0].rpartition('/')[2]
3120+ print """
3121+Usage: %s [options] --import-desktop master.desktop
3122+ [--export-gettext to-launchpad]
3123+ [--import-gettext from-launchpad]
3124+ [--export-desktop improved.desktop]
3125+ [--project-name somename]
3126+
3127+ Convert a desktop file to Gettext files and back
3128+
3129+ options could be:
3130+ -v | --verbose verbose mode
3131+ --import-desktop file master desktop file (mandatory)
3132+ --import-gettext dir GetText files to merge
3133+ --export-gettext dir merged GetText files
3134+ --export-desktop file improved desktop file
3135+ --project-name name project or source package name
3136+
3137+""" % appname
3138+
3139+if '__main__' == __name__:
3140+ sys.stdout = codecs.getwriter('utf8')(sys.stdout)
3141+ try:
3142+ opts, args = getopt.getopt(sys.argv[1:], "dhv",
3143+ [ "verbose", "project-name=",
3144+ "import-desktop=", "export-desktop=",
3145+ "import-gettext=", "export-gettext=" ])
3146+ except getopt.GetoptError, err:
3147+ print str(err)
3148+ usage()
3149+ sys.exit(2)
3150+
3151+ verbose = False
3152+ desktop_in = None
3153+ desktop_out = None
3154+ gettext_in = None
3155+ gettext_out = None
3156+ project_name = "misconfigured-project"
3157+ for o, a in opts:
3158+ if o in ("-v", "--verbose"):
3159+ verbose = True
3160+ elif o in ("-h", "--help"):
3161+ usage()
3162+ sys.exit()
3163+ elif o == "--project-name":
3164+ project_name = a
3165+ elif o == "--import-desktop":
3166+ desktop_in = a
3167+ elif o == "--export-desktop":
3168+ desktop_out = a
3169+ elif o == "--import-gettext":
3170+ gettext_in = a
3171+ elif o == "--export-gettext":
3172+ gettext_out = a
3173+
3174+ if desktop_in is None:
3175+ print "Error: --import-desktop is mandatory"
3176+ usage()
3177+ sys.exit(2)
3178+
3179+ df = DesktopFile(desktop_in, src_pkg = project_name, verbose = verbose)
3180+ if gettext_in is not None:
3181+ df.import_gettext_files(gettext_in)
3182+ if gettext_out is not None:
3183+ df.export_gettext_files(gettext_out)
3184+ if desktop_out is not None:
3185+ if desktop_in != desktop_out or df.changed:
3186+ df.write_desktop(desktop_out)
3187+ exit(1 if df.changed else 0)
3188
3189=== added file 'update-inspector.py'
3190--- update-inspector.py 1970-01-01 00:00:00 +0000
3191+++ update-inspector.py 2024-02-28 12:58:47 +0000
3192@@ -0,0 +1,149 @@
3193+#!/usr/bin/python
3194+# -*- coding: utf-8 -*-
3195+
3196+# (c) 2010, Fabien Tassin <fta@ubuntu.com>
3197+
3198+# Helper to merge the localizedStrings.js strings from Wekbit inspector into
3199+# the inspector_strings.grd Grit template
3200+
3201+import os, re, sys, codecs
3202+from optparse import OptionParser
3203+
3204+class JS2Grit:
3205+ def __init__(self, grd = None, js = None):
3206+ self.js_file = None
3207+ self.js_strings = []
3208+ self.grd_file = None
3209+ self.order = []
3210+ self.data = {}
3211+ self.missing = []
3212+ self.obsolete = []
3213+ self.merged = []
3214+ if grd is not None:
3215+ self.import_grd(grd)
3216+ if js is not None:
3217+ self.import_js(js)
3218+
3219+ def xml2js(self, s):
3220+ '''
3221+ '<ph name="ERRORS_COUNT">%1$d<ex>2</ex></ph> errors, <ph name="WARNING_COUNT">%2$d<ex>1</ex></ph> warning'
3222+ => '%d errors, %d warnings'
3223+ '''
3224+ s2 = re.sub('<ex>.*?</ex>', '', s)
3225+ s2 = re.sub('<ph name=".*?">%\d+\$(.*?)</ph>', r'%\1', s2)
3226+ s2 = re.sub('&lt;', '<', s2)
3227+ s2 = re.sub('&gt;', '>', s2)
3228+ s2 = re.sub('\'\'\'', '', s2)
3229+ return re.sub('<ph name=".*?">(.*?)</ph>', r'\1', s2)
3230+
3231+ def js2xml(self, s):
3232+ '''
3233+ '%d errors, %d warnings'
3234+ => '<ph name="XXX">%1$d<ex>XXX</ex></ph> errors, <ph name="XXX">%2$d<ex>XXX</ex></ph> warning'
3235+ '''
3236+ s = re.sub('<', '&lt;', s)
3237+ s = re.sub('>', '&gt;', s)
3238+ s = re.sub('^ ', '\'\'\' ', s)
3239+ phs = [ x for x in re.split(r'(%\d*\.?\d*[dfs])', s) if x.find('%') == 0 and x.find('%%') != 0 ]
3240+ if len(phs) > 1:
3241+ phs = re.split(r'(%\d*\.?\d*[dfs])', s)
3242+ j = 1
3243+ for i, part in enumerate(phs):
3244+ if part.find('%') == 0 and part.find('%%') != 0:
3245+ phs[i] = '<ph name="XXX">%%%d$%s<ex>XXX</ex></ph>' % (j, part[1:])
3246+ j += 1
3247+ elif len(phs) == 1:
3248+ phs = re.split(r'(%\d*\.?\d*[dfs])', s)
3249+ for i, part in enumerate(phs):
3250+ if part.find('%') == 0 and part.find('%%') != 0:
3251+ phs[i] = '<ph name="XXX">%s<ex>XXX</ex></ph>' % part
3252+ else:
3253+ return s
3254+ return ''.join(phs)
3255+
3256+ def import_grd(self, file):
3257+ self.order = []
3258+ self.data = {}
3259+ self.grd_file = file
3260+ fd = codecs.open(file, 'rb', encoding='utf-8')
3261+ file = fd.read()
3262+ for s in re.finditer('<message name="(.*?)" desc="(.*?)">\n\s+(.*?)\n\s+</message>', file, re.S):
3263+ key = self.xml2js(s.group(3))
3264+ self.order.append(key)
3265+ self.data[key] = { 'code': s.group(1), 'desc': s.group(2), 'string': s.group(3) }
3266+ fd.close()
3267+ return self.order, self.data
3268+
3269+ def import_js(self, file):
3270+ self.js_strings = []
3271+ self.js_file = file
3272+ fd = codecs.open(file, 'rb', encoding='utf-16')
3273+ file = fd.read()
3274+ for s in re.finditer('localizedStrings\["(.*?)"\] = "(.*?)";', file, re.S):
3275+ self.js_strings.append(s.group(1))
3276+ fd.close()
3277+ return self.js_strings
3278+
3279+ def merge_strings(self):
3280+ self.merged = []
3281+ self.missing = [ s for s in self.js_strings if s not in self.order ]
3282+ self.obsolete = [ s for s in self.order if s not in self.js_strings ]
3283+ for s in self.js_strings:
3284+ if s in self.order:
3285+ self.merged.append(self.data[s])
3286+ else:
3287+ self.merged.append({ 'code': 'IDS_XXX', 'desc': 'XXX', 'string': self.js2xml(s), 'key': s })
3288+
3289+ def get_new_strings_count(self):
3290+ return len(self.missing)
3291+
3292+ def get_obsolete_strings_count(self):
3293+ return len(self.obsolete)
3294+
3295+ def get_strings_count(self):
3296+ return len(self.js_strings)
3297+
3298+ def export_grd(self, grd):
3299+ fdi = codecs.open(self.grd_file, 'rb', encoding='utf-8')
3300+ data = fdi.read()
3301+ fdi.close()
3302+
3303+ fdo = codecs.open(grd, 'wb', encoding='utf-8')
3304+
3305+ # copy the header
3306+ pos = data.find('<messages')
3307+ pos += data[pos:].find('\n') + 1
3308+ fdo.write(data[:pos])
3309+
3310+ # write the merged strings
3311+ for s in self.merged:
3312+ if 'key' in s and s['key'] != s['string']:
3313+ fdo.write(" <!-- XXX: '%s' -->\n" % s['key'])
3314+ fdo.write(' <message name="%s" desc="%s">\n %s\n </message>\n' % \
3315+ (s['code'], s['desc'], s['string']))
3316+ # copy the footer
3317+ pos = data.find('</messages>')
3318+ pos -= pos - data[:pos].rfind('\n') - 1
3319+ fdo.write(data[pos:])
3320+ fdo.close()
3321+
3322+if '__main__' == __name__:
3323+ sys.stdout = codecs.getwriter('utf8')(sys.stdout)
3324+
3325+ parser = OptionParser(usage = 'Usage: %prog --grd inspector_strings.grd --js localizedStrings.js -o foo.grd')
3326+ parser.add_option("-j", "--js", dest="js",
3327+ help="read js strings from FILE", metavar="FILE")
3328+ parser.add_option("-g", "--grd", dest="grd",
3329+ help="read grd template from FILE", metavar="FILE")
3330+ parser.add_option("-o", "--output", dest="output",
3331+ help="write merged grd template to FILE", metavar="FILE")
3332+ (options, args) = parser.parse_args()
3333+
3334+ if options.grd is None or options.js is None or options.output is None:
3335+ parser.error("One of --grd, --js or --output is missing")
3336+ js2grd = JS2Grit(grd = options.grd, js = options.js)
3337+ print "Found %d strings in the js file" % js2grd.get_strings_count()
3338+ js2grd.merge_strings()
3339+ js2grd.export_grd(options.output)
3340+ print "Merged %d new strings, dropped %d obsolete strings" % \
3341+ (js2grd.get_new_strings_count(), js2grd.get_obsolete_strings_count())
3342
3343=== added file 'update-pot.sh'
3344--- update-pot.sh 1970-01-01 00:00:00 +0000
3345+++ update-pot.sh 2024-02-28 12:58:47 +0000
3346@@ -0,0 +1,87 @@
3347+#!/bin/sh
3348+
3349+# Update the gettext bzr branch (imported by lp rosetta)
3350+# based on a merge of all the templates in the 4 chromium
3351+# channels.
3352+# (c) 2010-2011, Fabien Tassin <fta@ubuntu.com>
3353+
3354+PROJECT=chromium-browser
3355+PKG_DIR=/data/bot/chromium-browser.head
3356+BIN_DIR=/data/bot/chromium-translations-tools.head
3357+OUT_DIR=/data/bot/upstream/chromium-translations.head
3358+LPE_DIR=/data/bot/upstream/chromium-translations-exports.head
3359+
3360+SRC_TRUNK_DIR=/data/bot/upstream/chromium-browser.svn/src
3361+SRC_DEV_DIR=/data/bot/upstream/chromium-dev.svn/src
3362+SRC_BETA_DIR=/data/bot/upstream/chromium-beta.svn/src
3363+SRC_STABLE_DIR=/data/bot/upstream/chromium-stable.svn/src
3364+
3365+######
3366+
3367+NEW_TEMPLATES="chrome/app/chromium_strings.grd,chrome/app/generated_resources.grd,ui/base/strings/ui_strings.grd,chrome/app/policy/policy_templates.grd,webkit/glue/inspector_strings.grd,webkit/glue/webkit_strings.grd"
3368+TEMPLATES="chrome/app/chromium_strings.grd,chrome/app/generated_resources.grd,ui/base/strings/app_strings.grd,chrome/app/policy/policy_templates.grd,webkit/glue/inspector_strings.grd,webkit/glue/webkit_strings.grd"
3369+
3370+OPTS="--map-template-names ui/base/strings/ui_strings.grd=ui/base/strings/app_strings.grd"
3371+IMPORT="--import-gettext $OUT_DIR,$LPE_DIR"
3372+BRANCH_TRUNK="--import-grit-branch trunk:$SRC_TRUNK_DIR:$NEW_TEMPLATES"
3373+BRANCH_DEV="--import-grit-branch dev:$SRC_DEV_DIR:$NEW_TEMPLATES"
3374+BRANCH_BETA="--import-grit-branch beta:$SRC_BETA_DIR:$NEW_TEMPLATES"
3375+BRANCH_STABLE="--import-grit-branch stable:$SRC_STABLE_DIR:$TEMPLATES"
3376+
3377+BRANCHES="$BRANCH_TRUNK $BRANCH_DEV $BRANCH_BETA $BRANCH_STABLE"
3378+
3379+(cd $LPE_DIR ; bzr pull -q)
3380+(cd $BIN_DIR ; bzr pull -q)
3381+
3382+cd $SRC_TRUNK_DIR
3383+$BIN_DIR/chromium2pot.py $BRANCHES $IMPORT $OPTS --export-gettext $OUT_DIR $NEW_TEMPLATES
3384+RET=$?
3385+cd $OUT_DIR
3386+set -e
3387+for f in */*.pot */*.po ; do
3388+ msgfmt -c $f
3389+done
3390+set +e
3391+rm -f messages.mo
3392+
3393+# desktop file
3394+DF_DIR="desktop_file"
3395+DF=$PROJECT.desktop
3396+DF_ARGS="-v --import-desktop $DF_DIR/$DF --project-name $PROJECT"
3397+if [ ! -d $DF_DIR ] ; then
3398+ mkdir $DF_DIR
3399+ RET=1
3400+fi
3401+if [ ! -e $DF_DIR/$DF ] ; then # no desktop file yet
3402+ cp -va $PKG_DIR/debian/$DF $DF_DIR
3403+ $BIN_DIR/desktop2gettext.py $DF_ARGS --export-gettext $DF_DIR
3404+ bzr add $DF_DIR
3405+ RET=1
3406+fi
3407+if [ -d $LPE_DIR/$DF_DIR ] ; then
3408+ $BIN_DIR/desktop2gettext.py $DF_ARGS --import-gettext $LPE_DIR/$DF_DIR --export-gettext $DF_DIR --export-desktop $DF_DIR/$DF
3409+ R=$?
3410+ if [ $R = 1 ] ; then
3411+ RET=1
3412+ fi
3413+ diff -u $PKG_DIR/debian/$DF $DF_DIR/$DF
3414+fi
3415+
3416+if [ $RET = 0 ] ; then
3417+ # no changes
3418+ exit 0
3419+fi
3420+
3421+REV=$(svn info $SRC_TRUNK_DIR | grep 'Last Changed Rev:' | cut -d' ' -f4)
3422+VERSION=$(cut -d= -f2 $SRC_TRUNK_DIR/chrome/VERSION | sed -e 's,$,.,' | tr -d '\n' | sed -e 's/.$//')
3423+MSG="Strings update for $VERSION r$REV"
3424+
3425+if [ "Z$1" != Z ] ; then
3426+ MSG=$1
3427+fi
3428+
3429+cd $OUT_DIR
3430+bzr add
3431+bzr commit -q -m "* $MSG"
3432+bzr push -q > /dev/null
3433+exit 0

Subscribers

People subscribed via source and target branches

to status/vote changes: