Setup notes for grid-updates

+ +

See README.html and the man page +for more information about grid-updates.

Requirements

Python version 2.6 and later. A Python installation is not required +for Windows users.

See README.html and the man page for more information about grid-updates.

Requirements

Python version 2.6 and later. Python is not a requirement for Windows users.

Installation

Users with an available Python installation can install grid-updates with the command python setup.py install from within the source archive.

Debian users can build packages with the command fakeroot debian/rules binary. This will build a package that can be installed using dpkg. This package will include the script and its documentation.

Installation

Users with an available Python installation can install grid-updates +with the command python setup.py install from within the source +archive.

Debian users can build packages with the command +fakeroot debian/rules binary. This will build a package that can be +installed using dpkg. This package will include the script and its +documentation.

Windows users have the following available options:

Binary installer for use on systems without Python (grid-updates-$ver.py2exe.exe)
Installer for systems with Python (grid-updates-$ver.win32.exe)
Installation from the source archive (python setup.py install as above)

Binary installer for use on systems without Python (grid-updates-1.2.0.py2exe.exe)
Installer for systems with Python (grid-updates-1.2.0.win32.exe)
Installation from the source archive (python setup.py install as above)

In short: if you're running Windows and don't know which file to get, download the py2exe version.

Consider running grid-updates --patch-tahoe after installation. This will apply a small patch to Tahoe-LAFS's web console to allow displaying news from grid-update's in an Iframe. The web console can be restored to its original state with grid-updates --undo-tahoe-patch.

Permissions

In order for this script to work, it needs to have read and write permission for the Tahoe-LAFS node's directory. This directory is typically located at %appdata%\.tahoe for Windows users and ~/.tahoe for other systems.

If asked to, grid-updates will update your introducers file and make a backup of it. When fetching NEWS, the script will write a plain text version to ~/.tahoe/NEWS. Additionally, an HTML version and an Atom news feed will be saved to your Tahoe node's web.static directory (typically ~/.tahoe/public_html). NEWS.html and NEWS.atom can be viewed in the node's /static directory, or (having patched Tahoe's Web UI) right on your node's front page

In short: if you're running Windows and don't know which file to get, download +the py2exe version.

Consider running grid-updates --patch-tahoe after installation. This will +apply a small patch to Tahoe-LAFS's web console to allow displaying news from +grid-updates in an Iframe. The web console can be restored to its original +state with grid-updates --undo-tahoe-patch.

Permissions

In order for this script to work, it needs to have read and write permission +for the Tahoe-LAFS node's directory. This directory is typically located at +%APPDATA%\.tahoe for Windows users and ~/.tahoe for other systems.

If asked to, grid-updates will update your introducers file and make a +backup of it. When fetching NEWS, the script will write a plain text version to +~/.tahoe/NEWS. Additionally, an HTML version and an Atom news feed will be +saved to your Tahoe node's web.static directory (typically +~/.tahoe/public_html). NEWS.html and NEWS.atom can be viewed in the node's +/static directory, or (having patched Tahoe's Web UI) right on your node's +front page

diff -Nru grid-updates-1.1.3/INSTALL.txt grid-updates-1.2.0/INSTALL.txt --- grid-updates-1.1.3/INSTALL.txt 2012-06-30 13:48:06.000000000 +0000 +++ grid-updates-1.2.0/INSTALL.txt 2012-09-09 12:14:59.000000000 +0000 @@ -2,13 +2,14 @@ Setup notes for grid-updates ============================ -See `README\.txt`_ and the `man page`_ +See `README.txt`_ and the `man page`_ for more information about grid-updates. Requirements ------------ -Python version 2.6 and later. Python is not a requirement for Windows users. +Python version 2.6 and later. A Python installation is not required +for Windows users. Installation ------------ @@ -34,7 +35,7 @@ Consider running ``grid-updates --patch-tahoe`` after installation. This will apply a small patch to Tahoe-LAFS's web console to allow displaying news from -grid-update's in an Iframe. The web console can be restored to its original +grid-updates in an Iframe. The web console can be restored to its original state with ``grid-updates --undo-tahoe-patch``. Permissions @@ -42,7 +43,7 @@ In order for this script to work, it needs to have read and write permission for the Tahoe-LAFS node's directory. This directory is typically located at -``%appdata%\.tahoe`` for Windows users and ``~/.tahoe`` for other systems. +``%APPDATA%\.tahoe`` for Windows users and ``~/.tahoe`` for other systems. If asked to, ``grid-updates`` will update your introducers file and make a backup of it. When fetching NEWS, the script will write a plain text version to @@ -52,7 +53,7 @@ `/static`_ directory, or (having patched Tahoe's Web UI) right on your node's `front page`_ -.. _README\.txt: README.txt +.. _README.txt: README.txt .. _man page: man/grid-updates.1.md .. _/static: http://127.0.0.1:3456/static/ .. _front page: http://127.0.0.1:3456/ diff -Nru grid-updates-1.1.3/MAN.html grid-updates-1.2.0/MAN.html --- grid-updates-1.1.3/MAN.html 2012-06-30 13:49:30.000000000 +0000 +++ grid-updates-1.2.0/MAN.html 2012-09-09 14:59:28.000000000 +0000 @@ -11,7 +11,7 @@

NAME

grid-updates - Helper Script for Tahoe-LAFS Nodes

@@ -31,10 +31,7 @@

Retrieve the news feed. See the NEWS section below.

-r, --repair

Maintain or repair the health of the subscription service's URIs.

-R, --repair-list

Retrieve a list of shares and maintain/repair them.

Maintain the health of Tahoe shares listed in a subscription.

--patch-tahoe

Patch the Tahoe-LAFS web console to display the grid-updates news feed in an Iframe.

@@ -43,7 +40,7 @@

Remove the grid-updates patch to the Tahoe web console and restore its original version.

--make-news file

Create a grid-updates-compatible NEWS.tgz file from a Markdown source file.

Create a grid-updates-compatible NEWS.tgz file from a reStructuredText source file.

--check-version

Check for a new version of this script on the grid.

@@ -63,20 +60,20 @@

-d directory, --node-directory directory

Specify the node directory (default: ~/.tahoe),

--format tar|deb|zip|arch|exe|py2exe

Specify in which format to download the update. Choices are: 'tar' (unix source archive), 'deb' (Debian package), 'arch' (ArchLinux package), 'zip' (Windows source archive), 'exe' (Windows installer [requires Python]), 'py2exe' (Windows installers [doesn't require Python]).

--format tar|deb|zip|exe|py2exe

Specify in which format to download the update. Choices are: 'tar' (unix source archive), 'deb' (Debian package), 'zip' (Windows source archive), 'exe' (Windows installer [requires Python]), 'py2exe' (Windows installer [doesn't require Python]).

--list-uri URI

--list-uri FILE CAP

Override the default location of the introducers list.

--news-uri URI

--news-uri FILE CAP

Override the default location of the NEWS.tgz file.

--script-uri URI

--script-uri DIR CAP

Override the default location of script updates.

--repairlist-uri URI

Override the default location of the --repair-list subscription file.

--repairlist-uri FILE CAP

Override the default location of the --repair subscription file.

-v

Increase verbosity of output.

@@ -96,32 +93,26 @@

tahoe_node_dir = directory

Same as --node-dir above

list_uri = URI

list_uri = FILE CAP

Same as --list-uri option above

news_uri = URI

news_uri = FILE CAP

Same as --news-uri option above

script_uri = URI

script_uri = DIR CAP

Same as --script-uri option above

comrepair_uri = URI

comrepair_uri = FILE CAP

Same as --comrepair-uri option above

NOTES

URIs, in this context, are Tahoe-LAFS directories like, for example URI:DIR2-RO:22s6zidugdxaeikq6lakbxbcci:mgrc3nfnygslyqrh7hds22usp6hbn3pulg5bu2puv6y3wpoaaqqq.

NEWS

If you choose to download the news feed, grid-updates will place a plain text version of it in your node's directory and print it to stdout. This is intended to be sent by cron mail.

There is also an HTML version of the news feed that you can view in a web browser. You can access it either directly (http://127.0.0.1:3456/static/NEWS.html) or have it be displayed on your Tahoe node's web console. You can prepare the console by running grid-updates --patch-tahoe once.

The Atom news feed (http://127.0.0.1:3456/static/NEWS.atom) can be used by regular feed readers to check for grid-updates news. (Please note, however, that you cannot "refresh" the feed with regular news readers. These files have to always be fetched by grid-updates first.)

- -

All grid-updates subscriptions reside on the Tahoe grid, which means that they need to be maintained (renewal of leases, repairs). Please contribute to their maintenance by running --repair from time to time.

If the script repeatedly fails to retrieve files from the grid, the share may be damaged and you will have to find a new set of URIs to subscribe to. One way to possibly get them is to run --check-version to see if there is a new version of grid-updates available. Newer versions might already include new default URIs.

INFORMATION FOR SUBSCRIPTION MAINTAINERS

If you want to offer a grid-updates subscription service, you will have to provide users with URIs to directories that contain the subscription files. The available subscription types are the news feed (called NEWS.tgz), the introducer list (called introducers.json.txt) and the repair list (called repair-list.json.txt).

If you want to offer a grid-updates subscription service, you will have to provide users with the URIs of your subscription files. They can be either in the form DIR CAP/filename or the the file cap itself. In the latter case you must make sure to create mutable files.

For the specific requirements of the JSON lists, please see the included example files in the share directory.

Please note that grid-updates expects the URIs of directories, not of the files themselves. It will append the filenames itself.

FILES

~/.tahoe/introducers
$XDG_CONFIG_DIRS/grid-updates/config (most commonly /etc/xdg)

BUGS

ERROR: Can't parse JSON list: No JSON object could be decoded

Currently Tahoe doesn't return JSON data if it encounters exceptions (see Trac ticket #1799). If you see this grid-updates error, you can rerun your command in debug mode (-vvv) to see Tahoe's actual response.

This error is most likely to occur during deep-check operations. If it does, it probably encountered the NotEnoughSharesError error, which means that a file was unrecoverable. You should investigate the problem using Tahoe directly.

ERROR: Could not run one-check for testfile: HTTP Error 410: Gone

This error is related to the one above but happens during one-check operations. If a file is not retrievable (due to not enough remaining shares) Tahoe responds with HTTP error 410.

Please report bugs in #tahoe-lafs on Irc2p or via email (see above).

Introduction to grid-updates

grid-updates is a Python script intended to help keep Tahoe-LAFS nodes' configurations up-to-date. It can retrieve lists of introducers ¹ as well as news feeds from the Tahoe grid. This is useful for any public grid that relies solely on volunteers.

On some public grids (especially the one on I2P) all nodes, even introducers, are run by volunteers and may disappear at any given time. Maintaining a list of all known introducers and distributing it to all participants of the grid will ensure the best possible connectivity for everyone.

Furthermore, there is no reliable way to contact node operators. This is why we want to encourage users to subscribe to a news feed relevant to their Tahoe grid. We hope it's going to be a way to inform unknown node operators about their wrongly configured nodes, necessary updates, recommended configuration changes and such.

Download

grid-updates can be downloaded from any of the following locations.

Project Eepsite
Tahoe grid ²
Git
-
- Git repository (Gitweb)
- Git mirror 1
- Git mirror 2 (Gitweb 2 )

+ + +

Introduction to grid-updates

grid-updates is a Python script intended to help keep Tahoe-LAFS nodes' +configurations up-to-date. It can retrieve lists of introducers [1] as well as +news feeds from the Tahoe grid. This is useful for any public grid that relies +solely on volunteers.

On some public grids (especially the one on I2P) all nodes, even +introducers, are run by volunteers and may disappear at any given time. +Maintaining a list of all known introducers and distributing it to all +participants of the grid will ensure the best possible connectivity for +everyone.

Furthermore, there is no reliable way to contact node operators. This is +why we want to encourage users to subscribe to a news feed relevant to +their Tahoe grid. We hope it's going to be a way to inform unknown node +operators about their wrongly configured nodes, necessary updates, +recommended configuration changes and such.

Download

grid-updates can be downloaded from any of the following locations.

Project Eepsite
Tahoe grid: URI:DIR2-RO:hgh5ylzzj6ey4a654ir2yxxblu:hzk3e5rbsefobeqhliytxpycop7ep6qlscmw4wzj5plicg3ilotq
Git
- Git repository (Gitweb)
- Git mirror 1
- Git mirror 2 (Gitweb 2 )
+

Installation

See INSTALL.html for information on how to install grid-updates.

Usage

Running the script

grid-updates was designed as a cron job script but can be run manually just as well.

Run this script with either --sync-introducers or --merge-introducers to make sure your Tahoe-LAFS node will know about as many introducers as possible.

If you also want to receive the news feed, add the --download-news action. It will fetch and display (email if run by a cron job) the news feed from the grid. It will also allow you to view the news your browser (using --patch-tahoe even in Tahoe's web console) and to be notified of news by a regular news reader using an Atom news feed. Following the grid-updates news is recommended.

The subscriptions and script installation files are stored on the grid itself and -- like all other shares -- need maintenance and repairs ³. If you can, please also add the --repair action to your cron job, or run it separately every once in a while. This is necessary to keep the service available.

--repair-list is another subscription-based service. It will download download a list of shares and check/repair them. This is supposed to make it easier for a community or group of people to support each others Tahoe shares. (The default subscription includes shares such as the DeepWiki on the I2P grid.)

Please refer to the man page for detailed usage information.

Example cron job setup

Installation

See INSTALL.html for information on how to install +grid-updates.

Usage

Running the script

grid-updates was designed as a cron job script but can be run +manually just as well.

Run this script with either --sync-introducers or +--merge-introducers to make sure your Tahoe-LAFS node will know +about as many introducers as possible.

If you also want to receive the news feed, add the --download-news +action. It will fetch and display (email if run by a cron job) the news +feed from the grid. It will also allow you to view the news your browser +(using --patch-tahoe even in Tahoe's web console) and to be notified +of news by a regular news reader using an Atom news feed. Following the +grid-updates news is recommended.

--repair is another subscription-based service. It will download a list of +shares and check/repair them. This is supposed to make it easier for a +community or group of people to support each other's Tahoe shares. (The +default subscription includes shares such as the DeepWiki on the I2P grid.)

Please refer to the man page for detailed usage information.

Example cron job setup

0  0 *   * * grid-updates --sync-introducers --download-news --check-version
-30 0 */2 * * grid-updates --repair

+0  0 *   * * grid-updates --sync-introducers --download-news --check-version
+30 0 */2 * * grid-updates --repair
+

Authors

darrob <darrob@mail.i2p>
KillYourTV <killyourtv@mail.i2p>

Authors

darrob <darrob@mail.i2p>
KillYourTV <killyourtv@mail.i2p>

License

grid-updates has been released into the public domain. You may do what you wish with it.

Support

For Bug reports and support join #tahoe-lafs on Irc2P or send an email.

You need at least Tahoe version 1.8.3 patched for multiple introducer support. See <http://killyourtv.i2p/tag/multiple_introducer/>↩
The Tahoe URI of the script is URI:DIR2-RO:hgh5ylzzj6ey4a654ir2yxxblu:hzk3e5rbsefobeqhliytxpycop7ep6qlscmw4wzj5plicg3ilotq↩
See also the tahoe-repair-all.sh script at <http://killyourtv.i2p/tahoe-lafs/scripts/>↩

License

grid-updates has been released into the public domain. You may do +what you wish with it.

Support

For Bug reports and support join +#tahoe-lafs on Irc2P or send an +email.

+ + + + + +

[1]	You need at least Tahoe version 1.8.3 patched for multiple introducer +support. See <http://killyourtv.i2p/tag/multiple_introducer/>

diff -Nru grid-updates-1.1.3/README.txt grid-updates-1.2.0/README.txt --- grid-updates-1.1.3/README.txt 2012-06-30 13:48:06.000000000 +0000 +++ grid-updates-1.2.0/README.txt 2012-09-09 12:14:59.000000000 +0000 @@ -1,13 +1,11 @@ Introduction to grid-updates ============================ -``grid-updates`` is a Python script intended to help keep `Tahoe\-LAFS`_ nodes' +``grid-updates`` is a Python script intended to help keep `Tahoe-LAFS`_ nodes' configurations up-to-date. It can retrieve lists of introducers [1]_ as well as news feeds from the Tahoe grid. This is useful for any public grid that relies solely on volunteers. -.. _Tahoe\-LAFS: http://www.tahoe-lafs.org - On some public grids (especially the one on I2P) all nodes, even introducers, are run by volunteers and may disappear at any given time. Maintaining a list of all known introducers and distributing it to all @@ -20,13 +18,16 @@ operators about their wrongly configured nodes, necessary updates, recommended configuration changes and such. +.. _Tahoe-LAFS: http://www.tahoe-lafs.org + + Download ======== ``grid-updates`` can be downloaded from any of the following locations. - `Project Eepsite`_ -- `Tahoe grid`_ [2]_ +- Tahoe grid: `URI:DIR2-RO:hgh5ylzzj6ey4a654ir2yxxblu:hzk3e5rbsefobeqhliytxpycop7ep6qlscmw4wzj5plicg3ilotq`_ - Git - `Git repository`_ (`Gitweb`_) @@ -34,17 +35,19 @@ - `Git mirror 2 `_ (`Gitweb 2`_ ) .. _Project Eepsite: http://killyourtv.i2p/grid-updates/ -.. _Tahoe grid: http://127.0.0.1:3456/uri/URI%3ADIR2-RO%3Ahgh5ylzzj6ey4a654ir2yxxblu%3Ahzk3e5rbsefobeqhliytxpycop7ep6qlscmw4wzj5plicg3ilotq/ +.. _`URI:DIR2-RO:hgh5ylzzj6ey4a654ir2yxxblu:hzk3e5rbsefobeqhliytxpycop7ep6qlscmw4wzj5plicg3ilotq`: http://127.0.0.1:3456/uri/URI%3ADIR2-RO%3Ahgh5ylzzj6ey4a654ir2yxxblu%3Ahzk3e5rbsefobeqhliytxpycop7ep6qlscmw4wzj5plicg3ilotq/ .. _Git repository: http://git.repo.i2p/r/grid-updates.git .. _Gitweb: http://git.repo.i2p/w/grid-updates.git .. _Gitweb 2: http://killyourtv.i2p/gitweb/?p=grid-updates.git + Installation ============ -See `INSTALL\.txt`_ for information on how to install +See `INSTALL.txt`_ for information on how to install grid-updates. + Usage ===== @@ -65,17 +68,10 @@ of news by a regular news reader using an Atom news feed. Following the grid-updates news is recommended. -The subscriptions and script installation files are stored on the grid itself -and -- like all other shares -- need maintenance and repairs [3]_. If you can, -please also add the ``--repair`` action to your cron job, or run -it separately every once in a while. This is necessary to keep the service -available. - -``--repair-list`` is another subscription-based service. It will download -download a list of shares and check/repair them. This is supposed to make it -easier for a community or group of people to support each others Tahoe shares. -(The default subscription includes shares such as the DeepWiki on the I2P -grid.) +``--repair`` is another subscription-based service. It will download a list of +shares and check/repair them. This is supposed to make it easier for a +community or group of people to support each other's Tahoe shares. (The +default subscription includes shares such as the DeepWiki on the I2P grid.) Please refer to the `man page`_ for detailed usage information. @@ -87,18 +83,21 @@ 0 0 * * * grid-updates --sync-introducers --download-news --check-version 30 0 */2 * * grid-updates --repair + Authors ======= - darrob - KillYourTV + License ======= ``grid-updates`` has been released into the public domain. You may do what you wish with it. + Support ======= @@ -110,13 +109,5 @@ You need at least Tahoe version 1.8.3 patched for multiple introducer support. See -.. [2] - The Tahoe URI of the script is - ``URI:DIR2-RO:hgh5ylzzj6ey4a654ir2yxxblu:hzk3e5rbsefobeqhliytxpycop7ep6qlscmw4wzj5plicg3ilotq`` - -.. [3] - See also the tahoe-repair-all.sh script at - - .. _man page: man/grid-updates.1.md -.. _INSTALL\.txt: INSTALL.txt +.. _INSTALL.txt: INSTALL.txt diff -Nru grid-updates-1.1.3/TODO grid-updates-1.2.0/TODO --- grid-updates-1.1.3/TODO 2012-06-03 17:25:20.000000000 +0000 +++ grid-updates-1.2.0/TODO 2012-09-09 12:13:11.000000000 +0000 @@ -3,9 +3,10 @@ * always make sure there is at least a NEWS.html stub in web.static? * is __patch_version__ a valid variable name? * don't hardcode default URIs; use a default config file instead? +* backups of WUI files (created by --patch-tahoe) create regular files even if source is originally a symlink * create grid-admin script - make patch actions part of this - make --make-news action part of this - - add option to add new introducers through g-u (and check syntax)? - - add statistical action (observe storage node and introducer uptimes) + - add option to add new introducers through g-u (and check syntax) + - add statistical action (observe storage node and introducer uptimes)? diff -Nru grid-updates-1.1.3/debian/changelog grid-updates-1.2.0/debian/changelog --- grid-updates-1.1.3/debian/changelog 2012-06-30 17:46:25.000000000 +0000 +++ grid-updates-1.2.0/debian/changelog 2012-09-09 20:29:48.000000000 +0000 @@ -1,3 +1,16 @@ +grid-updates (1.2.0-1precise1) precise; urgency=low + + * Upload to PPA + + -- Kill Your TV Sun, 09 Sep 2012 20:29:42 +0000 + +grid-updates (1.2.0-1) stable; urgency=low + + * New upstream release + * Add dependency on python-argparse + + -- Kill Your TV Sun, 09 Sep 2012 14:42:38 +0000 + grid-updates (1.1.3-1precise1) precise; urgency=low * Upload to PPA diff -Nru grid-updates-1.1.3/debian/control grid-updates-1.2.0/debian/control --- grid-updates-1.1.3/debian/control 2012-06-30 17:46:25.000000000 +0000 +++ grid-updates-1.2.0/debian/control 2012-09-09 20:29:48.000000000 +0000 @@ -7,10 +7,11 @@ Vcs-Git: http://killyourtv.i2p/git/grid-updates.git XS-Python-Version: >=2.6 Vcs-Browser: http://killyourtv.i2p/gitweb/?p=grid-updates.git +Homepage: http://killyourtv.i2p/grid-updates Package: grid-updates Architecture: all -Depends: ${misc:Depends}, ${python:Depends} +Depends: ${misc:Depends}, ${python:Depends}, python-argparse Recommends: i2p, i2p-tahoe-lafs Description: helper script for Tahoe-LAFS grid-updates attempts to keep Tahoe-LAFS nodes' configurations up-to-date and diff -Nru grid-updates-1.1.3/gridupdates/config.py grid-updates-1.2.0/gridupdates/config.py --- grid-updates-1.1.3/gridupdates/config.py 2012-06-03 17:25:20.000000000 +0000 +++ grid-updates-1.2.0/gridupdates/config.py 2012-09-09 12:14:59.000000000 +0000 @@ -1,5 +1,5 @@ from __future__ import print_function -import optparse +import argparse import os import platform import sys @@ -32,10 +32,10 @@ # 1. Default settings default_config = { 'tahoe_node_dir' : default_tahoe_node_dir, - 'list_uri' : 'URI:DIR2-RO:2vuiokc4wgzkxgqf3qigcvefqa:45gscpimazsm44eoeern54b5t2u4gpf7363odjhut255jxkajpqa', - 'news_uri' : 'URI:DIR2-RO:hx6754mru4kjn5xhda2fdxhaiu:hbk4u6s7cqfiurqgqcnkv2ckwwxk4lybuq3brsaj2bq5hzajd65q', + 'list_uri' : 'URI:DIR2-RO:2vuiokc4wgzkxgqf3qigcvefqa:45gscpimazsm44eoeern54b5t2u4gpf7363odjhut255jxkajpqa/introducers.json.txt', + 'news_uri' : 'URI:DIR2-RO:hx6754mru4kjn5xhda2fdxhaiu:hbk4u6s7cqfiurqgqcnkv2ckwwxk4lybuq3brsaj2bq5hzajd65q/NEWS.tgz', 'script_uri' : 'URI:DIR2-RO:hgh5ylzzj6ey4a654ir2yxxblu:hzk3e5rbsefobeqhliytxpycop7ep6qlscmw4wzj5plicg3ilotq', - 'repairlist_uri' : 'URI:DIR2-RO:ysxswonidme22ireuqrsrkcv4y:nqxg7ihxnx7eqoqeqoy7xxjmsqq6vzfjuicjtploh4k7mx6viz3a', + 'repairlist_uri' : 'URI:DIR2-RO:ysxswonidme22ireuqrsrkcv4y:nqxg7ihxnx7eqoqeqoy7xxjmsqq6vzfjuicjtploh4k7mx6viz3a/repair-list.json.txt', 'output_dir' : os.path.abspath(os.getcwd()) } return default_config @@ -126,137 +126,125 @@ default_config = parse_config_files(argv) - # 3. Optparse + # 3. Argparse # defaults to values from Configparser - parser = optparse.OptionParser() + parser = argparse.ArgumentParser() # actions - action_opts = optparse.OptionGroup( - parser, 'Actions', - 'These arguments control which actions will be executed.') - action_opts.add_option('-s', '--sync-introducers', + action_opts = parser.add_argument_group( + 'Actions', 'These arguments control which actions will be executed.') + action_opts.add_argument('-s', '--sync-introducers', action = 'store_true', dest = "sync", default = False, help = "Synchronize the local list of introducers with the " "subscription's.") - action_opts.add_option('-m', '--merge-introducers', + action_opts.add_argument('-m', '--merge-introducers', action = 'store_true', dest = "merge", default = False, help = 'Downloads and merges list of introducers into your ' 'local list.') - action_opts.add_option('-n', '--download-news', + action_opts.add_argument('-n', '--download-news', action = 'store_true', dest = "news", default = False, help = 'Downloads news feed.') - action_opts.add_option('-r', '--repair', + action_opts.add_argument('-r', '--repair', action = 'store_true', dest = "repair", default = False, - help = 'Run a deep-check and repair on all grid-updates shares.') - action_opts.add_option('-R', '--repair-list', - action = 'store_true', - dest = "repairlist", - default = False, - help = 'Retrieve a list of shares and maintain/repair them.') - action_opts.add_option('--community-repair', + help = 'Retrieve a list of shares and maintain/repair them. ' + 'Repairs the grid-updates subscriptions by default.') + action_opts.add_argument('-R', '--repair-list', action = 'store_true', dest = "deprecated", default = False, - help = 'This action is deprecated! Please use --repair-list instead.') - action_opts.add_option('--check-version', + help = 'This action is deprecated! Please use --repair instead.') + action_opts.add_argument('--check-version', action = 'store_true', dest = "check_version", default = False, help = 'Check for a new version of grid-updates.') - action_opts.add_option('--download-update', + action_opts.add_argument('--download-update', action = 'store_true', dest = "download_update", default = False, help = 'Download a new version of grid-updates.') - action_opts.add_option('--patch-tahoe', + action_opts.add_argument('--patch-tahoe', action = 'store_true', dest = "patch_ui", default = False, help = ('Patch the Tahoe Web UI to display grid-updates news ' 'in an IFrame.')) - action_opts.add_option('--undo-patch-tahoe', + action_opts.add_argument('--undo-patch-tahoe', action = 'store_true', dest = "undo_patch_ui", default = False, help = 'Restore the original Tahoe Web console files.') - action_opts.add_option('--make-news', + action_opts.add_argument('--make-news', action = 'store', - dest = "mknews_md_file", + dest = "news_source_file", help = 'Compile a grid-updates-compatible NEWS.tgz file from' ' a Markdown file.') - parser.add_option_group(action_opts) + parser.add_argument_group(action_opts) # options - other_opts = optparse.OptionGroup( - parser, 'Options', - 'These arguments can override various settings.') - other_opts.add_option('-d', '--node-directory', + other_opts = parser.add_argument_group( + 'Options', 'These arguments can override various settings.') + other_opts.add_argument('-d', '--node-directory', action = 'store', dest = "tahoe_node_dir", default = default_config['tahoe_node_dir'], help = 'Specify the Tahoe node directory.') - other_opts.add_option('-u', '--node-url', + other_opts.add_argument('-u', '--node-url', action = 'store', dest = 'tahoe_node_url', help = "Specify the Tahoe gateway node's URL.") - other_opts.add_option('--list-uri', + other_opts.add_argument('--list-uri', action = 'store', dest = 'list_uri', default = default_config['list_uri'], help = 'Override default location of introducers list.') - other_opts.add_option('--news-uri', + other_opts.add_argument('--news-uri', action = 'store', dest = 'news_uri', default = default_config['news_uri'], help = 'Override default location of news list.') - other_opts.add_option('--script-uri', + other_opts.add_argument('--script-uri', action = 'store', dest = 'script_uri', default = default_config['script_uri'], help = 'Override default location of script releases.') - other_opts.add_option('--repairlist-uri', + other_opts.add_argument('--repairlist-uri', action = 'store', dest = 'repairlist_uri', default = default_config['repairlist_uri'], help = ('Override default location of additional repair ' 'subscription.')) - other_opts.add_option('--comrepair-uri', - action = 'store_true', - dest = 'deprecated', - default = False, - help = ('This option is deprecated! Please use --repairlist-uri ' - 'instead.')) - other_opts.add_option('--format', + other_opts.add_argument('--format', action = 'store', dest = 'update_format', default = 'tar', help = ('Specify in which format to download the update.')) - other_opts.add_option('-o', '--output-dir', + other_opts.add_argument('-o', '--output-dir', action = 'store', dest = 'output_dir', default = default_config['output_dir'], help = ('Override default output directory (%s) for script ' 'update downloads and NEWS.tgz generation.' % os.getcwd())) - parser.add_option_group(other_opts) + parser.add_argument_group(other_opts) # remaining - parser.add_option('-v', + parser.add_argument('-v', action='count', dest='verbosity', default=1, help = 'Increase verbosity (-vv for debug mode).') - parser.add_option('-q', + parser.add_argument('-q', action='store_const', const=0, dest='verbosity', help = 'Turn off verbosity.') - parser.add_option('-V', '--version', + parser.add_argument('-V', '--version', dest = "version", action = "store_true", default = False, @@ -264,12 +252,12 @@ # Fake option: this option will not be accessed directly; --config is used # for ConfigParser (see above); this entry exists here for completeness of # --help - parser.add_option('-c', '--config', + parser.add_argument('-c', '--config', action = 'store', dest = 'config', help = 'Manually specify a configuration file.') # parse arguments - (opts, args) = parser.parse_args() + opts = parser.parse_args() # DEBUG if opts.verbosity > 2: @@ -283,7 +271,7 @@ 'repairlist_uri', 'output_dir']: print('%16s: %s' % (opt, vars(opts)[opt])) - print("DEBUG: Patch directory is", find_datadir()) + print("DEBUG: Patch directory is:", find_datadir()) # Temporar: abort on deprecated options if opts.deprecated: @@ -291,4 +279,4 @@ sys.argv[0], file=sys.stderr) sys.exit(1) - return (opts, args) + return opts diff -Nru grid-updates-1.1.3/gridupdates/functions.py grid-updates-1.2.0/gridupdates/functions.py --- grid-updates-1.1.3/gridupdates/functions.py 2012-06-29 20:13:25.000000000 +0000 +++ grid-updates-1.2.0/gridupdates/functions.py 2012-09-09 12:14:59.000000000 +0000 @@ -5,7 +5,6 @@ import imp import json import os -import platform import re import sys # Maybe this is better than try -> except? @@ -138,7 +137,7 @@ """Determine the location of the tahoe installation directory and included 'web' directory by parsing the tahoe web console.""" webconsole = urlopen(tahoe_node_url) - match = re.search(r'.*\ \'(.*__init__.py)', webconsole.read()) + match = re.search(r'.*\ \'(.*__init__.py)', webconsole.read().decode('utf8')) if match: tahoe_dir = os.path.dirname(match.group(1)) return tahoe_dir @@ -174,7 +173,7 @@ def get_tahoe_version(tahoe_node_url): """Determine Tahoe-LAFS version number from web console.""" webconsole = urlopen(tahoe_node_url) - match = re.search(r'allmydata-tahoe:\ (.*),', webconsole.read()) + match = re.search(r'allmydata-tahoe:\ (.*),', webconsole.read().decode('utf')) version = match.group(1) return version @@ -216,19 +215,38 @@ """Investigates a JSON list's validity.""" try: keys = json.loads(json_list).keys() - except ValueError: - print("ERROR: Can't parse JSON list.", file=sys.stderr) + except ValueError as ve: + print("ERROR: Can't parse JSON list:", ve) + if verbosity > 2: + print(json_list) + return False + except: + print("ERROR: JSON data is invalid (Unexpected Error).") + if verbosity > 2: + print(json_list) + return False + else: + if verbosity > 3: + print('DEBUG: JSON list seems to be valid. Found %d keys.' % + len(keys)) + return True + +def subscription_list_is_valid(json_list, verbosity=0): + """Investigates a share list's JSON validity.""" + if json_list_is_valid(json_list): + keys = json.loads(json_list).keys() + for uri in list(keys): + try: + json.loads(json_list)[uri]['name'] + except TypeError: + print("ERROR: Can't parse JSON list.", file=sys.stderr) + return False + if verbosity > 3: + print('DEBUG: Subscription list seems to be valid. ' + 'Found %d keys.' % len(keys)) + return True + else: return False - for uri in list(keys): - try: - json.loads(json_list)[uri]['name'] - except TypeError: - print("ERROR: Can't parse JSON list.", file=sys.stderr) - return False - if verbosity > 2: - print('DEBUG: JSON list seems to be valid. Found %d keys.' % - len(keys)) - return True def install_news_stub(web_static_dir): """Copy a placeholder NEWS.html file to Tahoe's web.static directory to diff -Nru grid-updates-1.1.3/gridupdates/grid_updates.py grid-updates-1.2.0/gridupdates/grid_updates.py --- grid-updates-1.1.3/gridupdates/grid_updates.py 2012-06-17 11:05:07.000000000 +0000 +++ grid-updates-1.2.0/gridupdates/grid_updates.py 2012-09-09 12:14:59.000000000 +0000 @@ -33,10 +33,7 @@ install_opener(opener) # Parse config files and command line arguments - (opts, args) = parse_args(sys.argv) - - if len(args) > 0: - print("WARN: %s doesn't take arguments." % sys.argv[0]) + opts = parse_args(sys.argv) # ACTION PARSING AND EXECUTION # ============================ @@ -73,10 +70,9 @@ and not opts.repair and not opts.check_version and not opts.download_update - and not opts.repairlist and not opts.patch_ui and not opts.undo_patch_ui - and not opts.mknews_md_file): + and not opts.news_source_file): print('ERROR: You need to specify an action. Please see %s --help.' % sys.argv[0], file=sys.stderr) sys.exit(2) @@ -130,7 +126,7 @@ sys.exit(1) if opts.verbosity > 2: - print("DEBUG: Tahoe node dir is", opts.tahoe_node_dir) + print("DEBUG: Tahoe node dir is:", opts.tahoe_node_dir) # Run actions # ----------- @@ -166,14 +162,14 @@ webui.run_action('patch', web_static_dir) elif opts.undo_patch_ui: webui.run_action('undo', web_static_dir) - if opts.mknews_md_file: + if opts.news_source_file: mknews = MakeNews(opts.verbosity) - mknews.run_action(opts.mknews_md_file, opts.output_dir) + mknews.run_action(opts.news_source_file, opts.output_dir) if opts.repair: - repairs.repair_action(uri_dict, opts.verbosity) - if opts.repairlist: - repairs.repairlist_action(tahoe_node_url, - uri_dict['repairlist'][1], opts.verbosity) + repairlist = repairs.RepairList(tahoe_node_url, + uri_dict['repairlist'][1], + opts.verbosity) + repairlist.run_action() if __name__ == "__main__": diff -Nru grid-updates-1.1.3/gridupdates/introducers.py grid-updates-1.2.0/gridupdates/introducers.py --- grid-updates-1.1.3/gridupdates/introducers.py 2012-06-03 17:25:20.000000000 +0000 +++ grid-updates-1.2.0/gridupdates/introducers.py 2012-09-09 12:14:59.000000000 +0000 @@ -5,7 +5,7 @@ from gridupdates.functions import tahoe_dl_file from gridupdates.functions import is_valid_introducer -from gridupdates.functions import json_list_is_valid +from gridupdates.functions import subscription_list_is_valid class Introducers(object): """This class implements the introducer list related functions of @@ -14,7 +14,7 @@ def __init__(self, nodedir, url, verbosity=0): self.verbosity = verbosity self.nodedir = nodedir - self.url = url + '/introducers.json.txt' + self.url = url if self.verbosity > 0: print("-- Updating introducers --") self.old_list = [] @@ -59,7 +59,7 @@ print("ERROR: Couldn't read introducer list.", file=sys.stderr) return - if not json_list_is_valid(new_list, self.verbosity): + if not subscription_list_is_valid(new_list, self.verbosity): return for uri in json.loads(new_list).keys(): name = json.loads(new_list)[uri]['name'] diff -Nru grid-updates-1.1.3/gridupdates/makenews.py grid-updates-1.2.0/gridupdates/makenews.py --- grid-updates-1.1.3/gridupdates/makenews.py 2012-06-03 17:25:20.000000000 +0000 +++ grid-updates-1.2.0/gridupdates/makenews.py 2012-09-09 12:13:11.000000000 +0000 @@ -24,31 +24,31 @@ # find template locations self.datadir = find_datadir() - def run_action(self, md_file, output_dir): + def run_action(self, src_file, output_dir): """Call this method to execute the desired action (--make-news). It will run the necessary methods.""" - copyfile(md_file, os.path.join(self.tempdir, 'NEWS')) - md_file = os.path.join(self.tempdir, 'NEWS') - html_file = self.compile_md(md_file) + copyfile(src_file, os.path.join(self.tempdir, 'NEWS')) + src_file = os.path.join(self.tempdir, 'NEWS') + html_file = self.compile_src(src_file) if not html_file: print('ERROR: Could not compile HTML version.', file=sys.stderr) else: atom_file = self.compile_atom() - include_list = [md_file, html_file, atom_file] + include_list = [src_file, html_file, atom_file] self.make_tarball(include_list, output_dir) remove_temporary_dir(self.tempdir, self.verbosity) - def compile_md(self, mdfile): + def compile_src(self, src_file): """Compile an HTML version of the Markdown source of NEWS; return the file path.""" - source = mdfile + source = src_file output_html = os.path.join(self.tempdir, 'NEWS.html') pandoc_tmplt = os.path.join(self.datadir, 'pandoc-template.html') try: subprocess.call(["pandoc", + "-r", "rst", "-w", "html", - "-r", "markdown", "--email-obfuscation", "none", "--template", pandoc_tmplt, "--output", output_html, diff -Nru grid-updates-1.1.3/gridupdates/news.py grid-updates-1.2.0/gridupdates/news.py --- grid-updates-1.1.3/gridupdates/news.py 2012-06-03 17:25:20.000000000 +0000 +++ grid-updates-1.2.0/gridupdates/news.py 2012-09-09 12:13:11.000000000 +0000 @@ -58,7 +58,7 @@ def download_news(self): """Download NEWS.tgz file to local temporary file.""" - url = self.url + '/NEWS.tgz' + url = self.url if self.verbosity > 1: print("INFO: Downloading news from the Tahoe grid.") if self.verbosity > 2: @@ -134,10 +134,10 @@ """Print the contents of the newly downloaded NEWS file in the temporary directory.""" with open(os.path.join(self.tempdir, 'NEWS'), 'r') as tempnews: - for line in tempnews.readlines(): - print(' | ' + line, end='') - print("The NEWS file (printed above) is located here: %s." % + print('Contents of updated NEWS file (%s):' % os.path.join(self.tahoe_node_dir, 'NEWS')) + for line in tempnews.readlines(): + print(line, end='') def install_files(self): """Copy extracted NEWS files to their intended locations.""" diff -Nru grid-updates-1.1.3/gridupdates/patchwebui.py grid-updates-1.2.0/gridupdates/patchwebui.py --- grid-updates-1.1.3/gridupdates/patchwebui.py 2012-06-29 20:13:25.000000000 +0000 +++ grid-updates-1.2.0/gridupdates/patchwebui.py 2012-09-09 12:13:11.000000000 +0000 @@ -2,7 +2,7 @@ from __future__ import print_function from distutils.version import LooseVersion -from shutil import copyfile +from shutil import copy2 import os import re import sys @@ -45,7 +45,7 @@ if mode == 'patch': if is_root(): print('WARN: Not installing NEWS.html placeholder (running' - ' as root.') + ' as root).') else: install_news_stub(web_static_dir) for uifile in list(self.filepaths.keys()): @@ -57,16 +57,32 @@ if self.verbosity > 1: print('INFO: A newer patch is available. ' 'Installing.') - self.install_file(uifile) + # Try to run function and abort if it fails. This + # will prevent g-u from trying the remaining files + # in the for loop. + # I'm not sure if this is the best way or a good + # idea at all. + if not self.install_file(uifile): + sys.exit(1) else: - if self.verbosity > 2: - print('DEBUG: Patch is up-to-date.') + if self.verbosity > 0: + print('Patch is up-to-date.') else: - self.backup_file(uifile) - self.install_file(uifile) + # Only try to install the patched uifile if the backup + # succeeded. + if self.backup_file(uifile): + self.install_file(uifile) + else: + sys.exit(1) if mode == 'undo': for uifile in list(self.filepaths.keys()): - self.restore_file(uifile) + # Try to run function and abort if it fails. This + # will prevent g-u from trying the remaining files + # in the for loop. + # I'm not sure if this is the best way or a good + # idea at all. + if not self.restore_file(uifile): + sys.exit(1) def patch_update_available(self): """Function to check if patched WebUI files have available updates.""" @@ -116,14 +132,16 @@ def add_target_filepaths(self): """Add locations of original web UI files to the location dict.""" for targetfile in list(self.filepaths.keys()): - filepath = (os.path.join(self.webdir, targetfile)) - if not os.path.exists(filepath): - # Tahoe 1.9.x moves tahoe.css to a 'static' subdirectory + # tahoe.css is located in web, not web/static/ in Tahoe < 1.9 + if self.tahoe_version < LooseVersion('1.9'): + filepath = (os.path.join(self.webdir, targetfile)) + elif targetfile == 'tahoe.css': filepath = (os.path.join(self.webdir, 'static', targetfile)) - if not os.path.exists(filepath): - print('ERROR: Could not find %s.' % - filepath, file=sys.stderr) - sys.exit(1) + else: + filepath = (os.path.join(self.webdir, targetfile)) + if not os.path.exists(filepath): + print('ERROR: Could not find %s.' % filepath, file=sys.stderr) + sys.exit(1) self.filepaths[targetfile].append(filepath) def read_patch_version(self, uifile): @@ -142,16 +160,23 @@ def backup_file(self, uifile): """Make a backup copy of file if it doesn't already exist.""" - # TODO exception targetfile = self.filepaths[uifile][1] backupfile = targetfile + '.grid-updates.original' if not os.path.exists(backupfile): if self.verbosity > 2: print('DEBUG: Backing up %s' % targetfile) - copyfile(targetfile, backupfile) + try: + copy2(targetfile, backupfile) + except IOError as exc: + print('ERROR: Cannot backup original WUI:', exc, file=sys.stderr) + else: + if self.verbosity > 2: + print('DEBUG: Backup successful.') + return True else: if self.verbosity > 2: print('DEBUG: Backup of %s already exists.' % targetfile) + return True def restore_file(self, uifile): @@ -161,7 +186,15 @@ backupfile = targetfile + '.grid-updates.original' if self.verbosity > 0: print('Restoring %s' % backupfile) - copyfile(backupfile, targetfile) + try: + copy2(backupfile, targetfile) + except IOError as exc: + print('ERROR: Cannot restore original WUI:', exc, file=sys.stderr) + else: + if self.verbosity > 2: + print('DEBUG: File sucessfully restored.') + os.remove(backupfile) # does this require a separate permission check? + return True def install_file(self, uifile): @@ -171,4 +204,9 @@ targetfile = self.filepaths[uifile][1] if self.verbosity > 0: print('Installing patched version of %s' % targetfile) - copyfile(patchedfile, targetfile) + try: + copy2(patchedfile, targetfile) + except IOError as exc: + print('ERROR: Cannot install patched WUI files:', exc, file=sys.stderr) + else: + return True diff -Nru grid-updates-1.1.3/gridupdates/repairs.py grid-updates-1.2.0/gridupdates/repairs.py --- grid-updates-1.1.3/gridupdates/repairs.py 2012-06-03 17:25:20.000000000 +0000 +++ grid-updates-1.2.0/gridupdates/repairs.py 2012-09-09 12:14:59.000000000 +0000 @@ -1,6 +1,7 @@ from __future__ import print_function import json import random +import re import sys # Maybe this is better than try -> except? if sys.version_info[0] == 2: @@ -18,6 +19,7 @@ from gridupdates.functions import is_literal_file from gridupdates.functions import tahoe_dl_file from gridupdates.functions import json_list_is_valid +from gridupdates.functions import subscription_list_is_valid def repair_share(sharename, repair_uri, mode, verbosity=0): """Run (deep-)checks including repair and add-lease on a Tahoe share; @@ -27,7 +29,8 @@ if mode == 'deep-check': params = urlencode({'t': 'stream-deep-check', 'repair': 'true', - 'add-lease': 'true'} + 'add-lease': 'true', + 'output': 'json'} ).encode('utf8') elif mode == 'one-check': params = urlencode({'t': 'check', @@ -46,7 +49,7 @@ except HTTPError as exc: print('ERROR: Could not run %s for %s: %s' % (mode, sharename, exc), file=sys.stderr) - return + return exc except URLError as urlexc: print("ERROR: %s while running %s for %s." % (urlexc, mode, sharename), file=sys.stderr) @@ -107,67 +110,119 @@ unhealthy += 1 return status, unhealthy -def repair_action(uri_dict, verbosity=0): - """Repair all (deep-check) Tahoe shares in a dictionary.""" - if verbosity > 0: - print("-- Repairing the grid-updates Tahoe shares. --") - mode = 'deep-check' - unhealthy = 0 - # shuffle() to even out chances of all shares to get repaired - # (Is this useful?) - sharelist = list(uri_dict.keys()) - random.shuffle(sharelist) - for sharename in sharelist: - repair_uri = uri_dict[sharename][1] - results = repair_share(sharename, repair_uri, mode, verbosity) - if not results: + +class RepairList(object): + """ + The --repair-list command. Repair all shares in the subscription file. + + This will download a json file and repair the included shares. The json + file must include the repair mode to be used. Known modes are one-check, + deep-check and level-check. + + level-check is a custom mode that tries to be a limited version of + deep-check. It will repair a directory structure as far as the configured + number of levels allows. + """ + + def __init__(self, tahoe_node_url, subscription_uri, verbosity=0): + self.verbosity = verbosity + self.tahoe_node_url = tahoe_node_url + self.subscription_uri = subscription_uri + if verbosity > 0: + print("-- Repairing Tahoe shares. --") + self.unhealthy = 0 + + def run_action(self): + shares = self.dl_sharelist() + sharelist = json.loads(shares).keys() + # shuffle() to even out chances of all shares to get repaired + random.shuffle(list(sharelist)) + for uri in sharelist: + sharename = json.loads(shares)[uri]['name'] + repair_uri = gen_full_tahoe_uri(self.tahoe_node_url, uri) + mode = json.loads(shares)[uri]['mode'] + if mode == 'deep-check': + self.deep_check(sharename, repair_uri, mode) + elif mode == 'one-check': + self.one_check(sharename, repair_uri, mode) + elif mode.startswith('level-check '): + self.level_check(sharename, repair_uri, mode) + else: + print("ERROR: Unknown repair mode: '%s'." % mode, file=sys.stderr) + return + if self.verbosity > 0: + print('Repairs have completed (unhealthy: %d).' % self.unhealthy) + + def dl_sharelist(self): + """ + Attempt to retrieve sharelist from the grid. If the sharelist is determined to + be valid, the sharelist is returned. + """ + shares = tahoe_dl_file(self.subscription_uri, self.verbosity).read().decode('utf8') + if subscription_list_is_valid(shares, self.verbosity): + return shares + else: + return + + def add_subdir_items(self, repair_uris, sharename): + """ + This function checks if a given item is a directory and adds its contents + to the directory. + """ + shareuri = repair_uris[sharename] + dir_req = urlopen(shareuri + '?t=json').read().decode('utf8') + if not json.loads(dir_req)[0] == 'dirnode': + if self.verbosity > 2: + print('DEBUG: Skipping %s' % sharename) + return repair_uris + for child in json.loads(dir_req)[1]['children']: + childname = sharename + '/' + child + if self.verbosity > 2: + print('DEBUG: Adding %s to repair list' % childname) + repair_uris[childname] = shareuri + '/' + child + return repair_uris + + def one_check(self, sharename, repair_uri, mode): + """Performs a shallow repair on 'repair_uri'""" + result = repair_share(sharename, repair_uri, mode, self.verbosity) + if re.match(r'^HTTP\ Error\ 410:\ Gone$', result.decode('utf8')): + status = 'not retrievable' + self.unhealthy += 1 + elif json_list_is_valid(result.decode('utf8'), self.verbosity): + status, self.unhealthy = parse_result(result.decode('utf8'), mode, + self.unhealthy, self.verbosity) + else: + status = 'unknown: Errors occured. Check this file manually to investigate.' + if self.verbosity > 1: + print(" Status: %s" % status) + + def deep_check(self, sharename, repair_uri, mode): + """Performs a deep recursive check on 'repair_uri'""" + results = repair_share(sharename, repair_uri, mode, self.verbosity) + if results is None: + print('WARN: Received no results.') return - if verbosity > 1: - print('INFO: Post-repair results for: %s' % sharename) for result in results: - status, unhealthy = parse_result(result.decode('utf8'), - mode, unhealthy, verbosity) - # Print summary - if unhealthy == 1: - sub = 'object' - else: - sub = 'objects' - if verbosity > 0: - print("Deep-check of grid-updates shares completed: " - "%d %s unhealthy." % (unhealthy, sub)) + if json_list_is_valid(result.decode('utf8'), self.verbosity): + status, self.unhealthy = parse_result(result.decode('utf8'), + mode, self.unhealthy, self.verbosity) -def repairlist_action(tahoe_node_url, subscription_uri, verbosity=0): - """The --repair-list command. Repair all shares in the subscription file.""" - if verbosity > 0: - print("-- Repairing Tahoe shares. --") - unhealthy = 0 - url = subscription_uri + '/repair-list.json.txt' - shares = tahoe_dl_file(url, verbosity).read().decode('utf8') - # create a list of URI's to be shuffled; also serves as syntax verification - if json_list_is_valid(shares): - sharelist = list(json.loads(shares).keys()) - else: - return - # shuffle() to even out chances of all shares to get repaired - random.shuffle(sharelist) - for uri in sharelist: - name = json.loads(shares)[uri]['name'] - repair_uri = gen_full_tahoe_uri(tahoe_node_url, uri) - mode = json.loads(shares)[uri]['mode'] - if mode == 'deep-check': - results = repair_share(name, repair_uri, mode, verbosity) - if not results: - return - for result in results: - status, unhealthy = parse_result(result.decode('utf8'), - mode, unhealthy, verbosity) - if mode == 'one-check': - result = repair_share(name, repair_uri, mode, verbosity) - if not result: - return - status, unhealthy = parse_result(result.decode('utf8'), mode, - unhealthy, verbosity) - if verbosity > 1: - print(" Status: %s" % status) - if verbosity > 0: - print('Repairs have completed (unhealthy: %d).' % unhealthy) + def level_check(self, sharename, repair_uri, mode): + """Performs a custom repair of 'repair_uri' %d levels deep.""" + levels = int(re.sub(r'level-check\ (\d+)', r'\1', mode)) + if self.verbosity > 1: + print('INFO: Will check %d levels deep.' % levels) + mode = 'one-check' # all item will be one-checked + repair_uris = {} + repair_uris[sharename] = repair_uri # add root dir + added = [] + while levels > 0: # keep adding items of subdirectories + for item in list(repair_uris.keys()): + if not item in added: + repair_uris = self.add_subdir_items(repair_uris, item) + added.append(item) + levels = levels - 1 + for item in sorted(list(repair_uris.keys())): + if self.verbosity > 2: + print('Will repair %s.' % sorted(list(repair_uris.keys()))) + self.one_check(item, repair_uris[item], mode) diff -Nru grid-updates-1.1.3/gridupdates/update.py grid-updates-1.2.0/gridupdates/update.py --- grid-updates-1.1.3/gridupdates/update.py 2012-06-26 10:18:20.000000000 +0000 +++ grid-updates-1.2.0/gridupdates/update.py 2012-09-09 12:13:11.000000000 +0000 @@ -54,8 +54,6 @@ download_filename = (basename + '.py2exe.exe') elif req_dist == 'exe': download_filename = (basename + '.win32.exe') - elif req_dist == 'arch': - download_filename = (basename + '-1-any.pkg.tar.xz') elif req_dist == 'deb': download_filename = ('grid-updates_' + self.latest_version + diff -Nru grid-updates-1.1.3/gridupdates/version.py grid-updates-1.2.0/gridupdates/version.py --- grid-updates-1.1.3/gridupdates/version.py 2012-06-30 13:49:20.000000000 +0000 +++ grid-updates-1.2.0/gridupdates/version.py 2012-09-09 14:42:23.000000000 +0000 @@ -3,5 +3,5 @@ __maintainer__ = ['darrob', 'KillYourTV'] __email__ = ['darrob@mail.i2p', 'killyourtv@mail.i2p'] __status__ = "Development" -__version__ = "1.1.3" -__patch_version__ = '1.8.3-gu6' +__version__ = "1.2.0" +__patch_version__ = 'gu7' diff -Nru grid-updates-1.1.3/man/grid-updates.1 grid-updates-1.2.0/man/grid-updates.1 --- grid-updates-1.1.3/man/grid-updates.1 2012-06-30 13:04:56.000000000 +0000 +++ grid-updates-1.2.0/man/grid-updates.1 2012-09-09 14:59:28.000000000 +0000 @@ -1,4 +1,4 @@ -.TH GRID-UPDATES 1 "April 2012" "User Commands" +.TH GRID-UPDATES 1 "September 2012" "User Commands" .SH NAME .PP grid-updates - Helper Script for Tahoe-LAFS Nodes @@ -33,12 +33,7 @@ .RE .TP .B -r, --repair -Maintain or repair the health of the subscription service\[aq]s URIs. -.RS -.RE -.TP -.B -R, --repair-list -Retrieve a list of shares and maintain/repair them. +Maintain the health of Tahoe shares listed in a subscription. .RS .RE .TP @@ -55,8 +50,8 @@ .RE .TP .B --make-news \f[I]file\f[] -Create a \f[I]grid-updates\f[]-compatible NEWS.tgz file from a Markdown -source file. +Create a \f[I]grid-updates\f[]-compatible NEWS.tgz file from a +reStructuredText source file. .RS .RE .TP @@ -88,32 +83,32 @@ .RS .RE .TP -.B --format \f[I]tar|deb|zip|arch|exe|py2exe\f[] +.B --format \f[I]tar|deb|zip|exe|py2exe\f[] Specify in which format to download the update. Choices are: \[aq]tar\[aq] (unix source archive), \[aq]deb\[aq] (Debian -package), \[aq]arch\[aq] (ArchLinux package), \[aq]zip\[aq] (Windows -source archive), \[aq]exe\[aq] (Windows installer [requires Python]), -\[aq]py2exe\[aq] (Windows installers [doesn\[aq]t require Python]). +package), \[aq]zip\[aq] (Windows source archive), \[aq]exe\[aq] (Windows +installer [requires Python]), \[aq]py2exe\[aq] (Windows installer +[doesn\[aq]t require Python]). .RS .RE .TP -.B --list-uri \f[I]URI\f[] +.B --list-uri \f[I]FILE CAP\f[] Override the default location of the introducers list. .RS .RE .TP -.B --news-uri \f[I]URI\f[] +.B --news-uri \f[I]FILE CAP\f[] Override the default location of the NEWS.tgz file. .RS .RE .TP -.B --script-uri \f[I]URI\f[] +.B --script-uri \f[I]DIR CAP\f[] Override the default location of script updates. .RS .RE .TP -.B --repairlist-uri \f[I]URI\f[] -Override the default location of the --repair-list subscription file. +.B --repairlist-uri \f[I]FILE CAP\f[] +Override the default location of the --repair subscription file. .RS .RE .TP @@ -146,30 +141,25 @@ .RS .RE .TP -.B list_uri = \f[I]URI\f[] +.B list_uri = \f[I]FILE CAP\f[] Same as --list-uri option above .RS .RE .TP -.B news_uri = \f[I]URI\f[] +.B news_uri = \f[I]FILE CAP\f[] Same as --news-uri option above .RS .RE .TP -.B script_uri = \f[I]URI\f[] +.B script_uri = \f[I]DIR CAP\f[] Same as --script-uri option above .RS .RE .TP -.B comrepair_uri = \f[I]URI\f[] +.B comrepair_uri = \f[I]FILE CAP\f[] Same as --comrepair-uri option above .RS .RE -.SH NOTES -.PP -\f[I]URIs\f[], in this context, are Tahoe-LAFS directories like, for -example -\f[C]URI:DIR2-RO:22s6zidugdxaeikq6lakbxbcci:mgrc3nfnygslyqrh7hds22usp6hbn3pulg5bu2puv6y3wpoaaqqq\f[]. .SH NEWS .PP If you choose to download the news feed, \f[I]grid-updates\f[] will @@ -190,35 +180,16 @@ (Please note, however, that you cannot "refresh" the feed with regular news readers. These files have to always be fetched by \f[I]grid-updates\f[] first.) -.SH SHARE HEALTH -.PP -All \f[I]grid-updates\f[] subscriptions reside on the Tahoe grid, which -means that they need to be maintained (renewal of leases, repairs). -Please contribute to their maintenance by running \f[C]--repair\f[] from -time to time. -.PP -If the script repeatedly fails to retrieve files from the grid, the -share may be damaged and you will have to find a new set of URIs to -subscribe to. -One way to possibly get them is to run \f[C]--check-version\f[] to see -if there is a new version of \f[I]grid-updates\f[] available. -Newer versions might already include new default URIs. .SH INFORMATION FOR SUBSCRIPTION MAINTAINERS .PP If you want to offer a \f[I]grid-updates\f[] subscription service, you -will have to provide users with URIs to directories that contain the -subscription files. -The available subscription types are the news feed (called -\f[B]NEWS.tgz\f[]), the introducer list (called -\f[B]introducers.json.txt\f[]) and the repair list (called -\f[B]repair-list.json.txt\f[]). +will have to provide users with the URIs of your subscription files. +They can be either in the form \f[I]DIR CAP/filename\f[] or the the file +cap itself. +In the latter case you must make sure to create mutable files. .PP For the specific requirements of the JSON lists, please see the included example files in the share directory. -.PP -Please note that \f[I]grid-updates\f[] expects the URIs of directories, -not of the files themselves. -It will append the filenames itself. .SH FILES .IP \[bu] 2 \f[I]~/.tahoe/introducers\f[] @@ -248,6 +219,30 @@ .IP \[bu] 2 \f[I]$XDG_CONFIG_DIRS/grid-updates/config\f[] (most commonly /etc/xdg) .SH BUGS +.TP +.B \f[B]ERROR: Can\[aq]t parse JSON list: No JSON object could be +decoded\f[] +Currently Tahoe doesn\[aq]t return JSON data if it encounters exceptions +(see Trac ticket #1799). +If you see this \f[B]grid-updates\f[] error, you can rerun your command +in debug mode (\f[C]-vvv\f[]) to see Tahoe\[aq]s actual response. +.RS +.PP +This error is most likely to occur during \f[I]deep-check\f[] +operations. +If it does, it probably encountered the \f[I]NotEnoughSharesError\f[] +error, which means that a file was unrecoverable. +You should investigate the problem using Tahoe directly. +.RE +.TP +.B \f[B]ERROR: Could not run one-check for testfile: HTTP Error 410: +Gone\f[] +This error is related to the one above but happens during +\f[I]one-check\f[] operations. +If a file is not retrievable (due to not enough remaining shares) Tahoe +responds with HTTP error 410. +.RS +.RE .PP Please report bugs in #tahoe-lafs on Irc2p or via email (see below). .SH SEE ALSO diff -Nru grid-updates-1.1.3/man/grid-updates.1.md grid-updates-1.2.0/man/grid-updates.1.md --- grid-updates-1.1.3/man/grid-updates.1.md 2012-06-26 10:18:20.000000000 +0000 +++ grid-updates-1.2.0/man/grid-updates.1.md 2012-09-09 12:14:59.000000000 +0000 @@ -1,6 +1,6 @@ % GRID-UPDATES(1) User Commands % darrob , KillYourTV -% April 2012 +% September 2012 NAME ==== @@ -33,10 +33,7 @@ : Retrieve the news feed. See the **NEWS** section below. -r, \--repair -: Maintain or repair the health of the subscription service's URIs. - --R, \--repair-list -: Retrieve a list of shares and maintain/repair them. +: Maintain the health of Tahoe shares listed in a subscription. \--patch-tahoe : Patch the Tahoe-LAFS web console to display the grid-updates news feed in @@ -47,8 +44,8 @@ original version. \--make-news *file* -: Create a *grid-updates*-compatible NEWS.tgz file from a Markdown source - file. +: Create a *grid-updates*-compatible NEWS.tgz file from a reStructuredText + source file. \--check-version : Check for a new version of this script on the grid. @@ -69,23 +66,23 @@ -d *directory*, \--node-directory *directory* : Specify the node directory (default: *~/.tahoe*), -\--format *tar|deb|zip|arch|exe|py2exe* +\--format *tar|deb|zip|exe|py2exe* : Specify in which format to download the update. Choices are: 'tar' (unix source - archive), 'deb' (Debian package), 'arch' (ArchLinux package), 'zip' (Windows source archive), + archive), 'deb' (Debian package), 'zip' (Windows source archive), 'exe' (Windows installer [requires Python]), - 'py2exe' (Windows installers [doesn't require Python]). + 'py2exe' (Windows installer [doesn't require Python]). -\--list-uri *URI* +\--list-uri *FILE CAP* : Override the default location of the introducers list. -\--news-uri *URI* +\--news-uri *FILE CAP* : Override the default location of the NEWS.tgz file. -\--script-uri *URI* +\--script-uri *DIR CAP* : Override the default location of script updates. -\--repairlist-uri *URI* -: Override the default location of the \--repair-list subscription file. +\--repairlist-uri *FILE CAP* +: Override the default location of the \--repair subscription file. -v : Increase verbosity of output. @@ -109,21 +106,15 @@ tahoe\_node\_dir = *directory* : Same as \--node-dir above -list_uri = *URI* +list_uri = *FILE CAP* : Same as \--list-uri option above -news_uri = *URI* +news_uri = *FILE CAP* : Same as \--news-uri option above -script_uri = *URI* +script_uri = *DIR CAP* : Same as \--script-uri option above -comrepair_uri = *URI* +comrepair_uri = *FILE CAP* : Same as \--comrepair-uri option above -NOTES -===== - -*URIs*, in this context, are Tahoe-LAFS directories like, for example -`URI:DIR2-RO:22s6zidugdxaeikq6lakbxbcci:mgrc3nfnygslyqrh7hds22usp6hbn3pulg5bu2puv6y3wpoaaqqq`. - NEWS ==== @@ -142,34 +133,17 @@ that you cannot "refresh" the feed with regular news readers. These files have to always be fetched by *grid-updates* first.) -SHARE HEALTH -============ - -All *grid-updates* subscriptions reside on the Tahoe grid, which means that -they need to be maintained (renewal of leases, repairs). Please contribute to -their maintenance by running `--repair` from time to time. - -If the script repeatedly fails to retrieve files from the grid, the share may -be damaged and you will have to find a new set of URIs to subscribe to. One -way to possibly get them is to run `--check-version` to see if there is a new -version of *grid-updates* available. Newer versions might already include new -default URIs. - INFORMATION FOR SUBSCRIPTION MAINTAINERS ======================================== If you want to offer a *grid-updates* subscription service, you will have to -provide users with URIs to directories that contain the subscription files. The -available subscription types are the news feed (called **NEWS.tgz**), the -introducer list (called **introducers.json.txt**) and the repair list (called -**repair-list.json.txt**). +provide users with the URIs of your subscription files. They can be either in +the form *DIR CAP/filename* or the the file cap itself. In the latter case you +must make sure to create mutable files. For the specific requirements of the JSON lists, please see the included example files in the share directory. -Please note that *grid-updates* expects the URIs of directories, not of the -files themselves. It will append the filenames itself. - FILES ===== @@ -183,8 +157,24 @@ BUGS ==== +**ERROR: Can't parse JSON list: No JSON object could be decoded** +: Currently Tahoe doesn't return JSON data if it encounters exceptions (see + Trac ticket #1799). If you see this **grid-updates** error, you can rerun + your command in debug mode (`-vvv`) to see Tahoe's actual response. + + This error is most likely to occur during *deep-check* operations. If it + does, it probably encountered the *NotEnoughSharesError* error, which means + that a file was unrecoverable. You should investigate the problem using + Tahoe directly. + +**ERROR: Could not run one-check for testfile: HTTP Error 410: Gone** +: This error is related to the one above but happens during *one-check* + operations. If a file is not retrievable (due to not enough remaining + shares) Tahoe responds with HTTP error 410. + Please report bugs in #tahoe-lafs on Irc2p or via email (see below). + SEE ALSO ======== diff -Nru grid-updates-1.1.3/share/tahoe.css.patched grid-updates-1.2.0/share/tahoe.css.patched --- grid-updates-1.1.3/share/tahoe.css.patched 2012-06-03 17:25:20.000000000 +0000 +++ grid-updates-1.2.0/share/tahoe.css.patched 2012-09-09 12:13:11.000000000 +0000 @@ -190,4 +190,4 @@ border: 1px solid #a00; padding: 2px } -/* grid-updates patch VERSION=1.8.3-gu6 */ +/* grid-updates patch VERSION=7 */ diff -Nru grid-updates-1.1.3/share/tahoe.css.patched19 grid-updates-1.2.0/share/tahoe.css.patched19 --- grid-updates-1.1.3/share/tahoe.css.patched19 2012-06-03 17:25:20.000000000 +0000 +++ grid-updates-1.2.0/share/tahoe.css.patched19 2012-09-09 12:13:11.000000000 +0000 @@ -191,4 +191,4 @@ border: 1px solid #a00; padding: 2px } -/* grid-updates patch VERSION=1.8.3-gu6 */ +/* grid-updates patch VERSION=7 */ diff -Nru grid-updates-1.1.3/share/welcome.xhtml.patched grid-updates-1.2.0/share/welcome.xhtml.patched --- grid-updates-1.1.3/share/welcome.xhtml.patched 2012-06-03 17:25:20.000000000 +0000 +++ grid-updates-1.2.0/share/welcome.xhtml.patched 2012-09-09 12:13:11.000000000 +0000 @@ -28,15 +28,15 @@

Setup notes for grid-updates

Requirements

Requirements

Installation

Installation

Permissions

Permissions

GRID-UPDATES(1) User Commands

darrob darrob@mail.i2p, KillYourTV killyourtv@mail.i2p

April 2012

September 2012

NAME

NOTES

NEWS

INFORMATION FOR SUBSCRIPTION MAINTAINERS

FILES

BUGS

SEE ALSO

Introduction to grid-updates

Download

Introduction to grid-updates

Download

Installation

Usage

Running the script

Example cron job setup

Installation

Usage

Running the script

Example cron job setup

Authors

Authors

License

Support

License

Support

Grid News