A cute introduction to Debtags

Facets

Facets are the aspects of our packages that we choose to categorise. Example of facets in Debtags are implemented-in, culture, interface, media.

Each facet defines an aspect of packages we look at. For each facet there is a set of categories that we use to describe what we see when we look at that aspect of a package.

Faceted categorization is like saying, "If I look at what is the use of this package, I see editing". "If I look at what is its user interface, I see a text-mode user interface".

Facets are "If I look at X"; tags are "I see X".

The list of facets currently categorised by Debtags is available in The Vocabulary.

Tags

The tag is a Debtags category. Debtags uses different sets of categories, that we call tags, to categorise different facets of software. We use the word tags instead of categories because it's shorter, and we talk about tags a lot.

In Debtags, we represent tags with a short English name, prefixed by the name of the facet it categorises. For example, when we look at what media types a package can work with and we categorise "The Gimp", we use the tag media::rasterimage.

The Vocabulary

The Vocabulary is where we store the list of facets and tags that we use for Debian packages.

Having a common vocabulary is important so that we all refer to the same property with the same name. Other benefits of the common vocabulary include the abilities to present selections of facets and tags in a user interface, to store meta-information about them (such as a short and long description, possibly in many languages), and to validate classification information.

The Debtags vocabulary is the file that delineates the facets and lists the respective tags that can be used to categorize Debian packages. The vocabulary file contains whatever needs to be stored about facets and tags. At the moment this means:

• a list of facets and their tags;
• description of facets and tags;
• editorial information;
• comments.

This information is stored in a format similar to the usual Debian package records. For example:

Facet: implemented-in
Description: Language that the package is implemented in
Nature: matter
Status: complete

Tag: implemented-in::c++
Description: C++

This defines a facet called implemented-in and a tag c++ inside it.

The main location of the vocabulary is in the subversion repository at svn://svn.debian.org/debtags/vocabulary/debian-packages; from there, it makes its way to the default Debtags tag source and then, by means of debtags update, it is stored locally in /var/lib/debtags/vocabulary. It is also indexed by debtags update to allow applications to quickly lookup entries inside it.

There are currently 32 facets defined, containing a total of 440 tags. To see the facets along with their descriptions and other data, you can run debtags update and then use grep-dctrl:

grep-dctrl -FFacet -r . /var/lib/debtags/vocabulary

A tutorial on sending patches to the vocabulary can be found at http://debtags.alioth.debian.org/vocabulary.html

The tag database

A tag is assigned to a package by entering the appropriate information in the Tag Database.

Designing this database is one of the most tricky and important implementation details of Debtags, as it is the way to share the tags and make them useful.

An obvious place for storing the tags is in the control file of the packages, together with the other package metadata. This possibility has been discarded, however, for a couple of good reasons:

• The aggregated package database is already big enough, and there are no clear short term ways of addressing this problem
• "Debian is a living organism": tags can change fairly often, because the classification is refined, a new tag is introduced or a whole new facet of packages is classified, and we cannot afford a new package upload or ftpmaster intervention every time such a change happens.

We chose instead to store tag data in a central tag repository, located at http://debian.vitavonni.de/packagebrowser, from which it can be downloaded in the user's computer by means of the debtags update command, just like the package database can be downloaded by means of apt-get update.

This makes it possible to store and maintain the categorization data without causing an added burden to the Debian infrastructure. It also allows to have different tag sources merged together. Check this out because it's cool!

Let's make an example with a fictitious Debian-Circus CDD (Custom Debian Distribution) that chooses to categorise packages also from the aspect ("facet") of their possible use in a circus. To do so, they want to have in Debtags a circus facet, categorised with tags such as travel-info, installation-procedures, inventory, artist-management, show-management, propaganda, accounting.

To realise this, Debian-Circus can create their own piece of Vocabulary and Tag Database and make it available to be downloaded as a tag source. debtags is able to download data from various tag sources (listed in /etc/debtags/sources.list) and merge them together.

Different tag sources can be merged easily when facets differ, and they complete each other's view of packages by shedding light on more aspects of them.

Now, Debian-Circus provides categorization along the circus facet, and Debian provides categorization for the suite and media facets. Any circus sysadmin can now merge this information together and look for circus::propaganda software for suite::gnome that can work with media::vectorimage.

Isn't it exciting?

The main location of the database is inside Erich Schubert's packagebrowser at http://debian.vitavonni.de/packagebrowser. From there, it makes its way to the default Debtags tag source and then, by means of debtags update, it is merged with other tag sources and stored locally in /var/lib/debtags/package-tags.

The database is stored locally in an easily parsable format that can also be understood by tools such as tagcoll and tagcolledit. It is also indexed by debtags update to allow applications to look entries up quickly in it.

The individual user can modify the Debtags database locally. Such local modifications are stored as tag patches in the file ~/.debtags/patch in the user's home directory. A tag patch is a special patch format that can represent a change to a tag database, and can also be applied to future versions of the database, so that your changes are preserved across debtags update invocations.

Another useful feature of the tag patches is that you can send them to the central database, which will integrate them with the rest of the data. You can do this by using debtags-edit (it has a "File/Mail changes to central database" feature) or by using the command debtags send.

Running debtags-edit

Try running debtags-edit. If it does not work, it might be because you have not yet run debtags update to download the Debtags data: do it now and run debtags-edit again.

Searching packages

On the left-hand side of the application window there is a Filter area which allows you to look for packages: you play with the controls, and a list of matching packages appears in the bottom left part of the screen. A notable part of the filter area is the Tags area, which lets you add tags to the filter.

Try pushing the Add button of the tags area and add something like "Use/Editing": the list will display all packages which can edit something.

Now push "Add" again: the list of facets and tags has narrowed down to only the available possibilities, and if you look inside the "Media" facet, you'll see a list of the possible kinds of media that can be edited using Debian packages.

Using the filter can search packages by name, do a full-text search à-la apt-cache search, search by maintainer, installed status or tags. This makes debtags-edit an interesting tool already, but let's see more.

[1]	If debtags-edit cannot find the DEBEMAIL environment variable, it will instead leave the "Maintainer:" field empty and preselect "Status: Installed", kindly suggesting you to check if the packages you know and use everyday are tagged well enough.

Tagging packages

You can now click on a package name in the bottom-left part of the application: that will display the package in the main display on the right side. At the bottom part you will see all the package data, very much like the output of apt-cache show, while in the top part you will see the tags of the package. If the package you chose has no tags, try looking for debtags: that will be nicely tagged.

Now try adding a new tag to debtags: try adding "Games/Toys". Push the other "Add" button below the tag list in the top-right part of the window, and navigate the complete facet list until you find the "Games" facet: inside it you will find "Toys". Select it: the tag has been added, and you made your first step into Debtags tagging!

Now try doing "File/Save", then go to a terminal and type debtags show debtags: you will notice that your new games::toys tag is already there. Cool, isn't it?

What happened is that debtags-edit saved your modifications in ~/.debtags/patch, and all Debtags-aware applications read that file at startup. Have a look at the file: it's just a patch with the change you made, and its format is such that it can be applied to any future version of the tag database. Isn't it smart? I'm so proud of it!

Now go back to debtags-edit and look in the "File" menu: there's a "Mail changes to central archive" option. If you click on it, it will mail the contents of your ~/.debtags/patch to the central archive, and the next day your contribute will be available to everyone when they will do debtags update.

What else do you need to contribute to the categorization? Only one more thing maybe: subscribe to the debtags-devel mailing list.

Using debtags-edit

These are some examples on how to use debtags-edit to do some more serious categorization work. Most of them are ways of identifying packages that need some work: once you find them in debtags-edit, you can directly work on them

1. Fire up debtags-edit and set the filter to display all installed packages with the tag special::not-yet-tagged. Now look at the results, pick something you know and add tags to it. After you are satisfied with its categorization, remove the not-yet-tagged tags and submit.

2. Click on "Edit/TODO dialog": a dialog will open with various TODO options:

   Empty tagset

   lists all packages with no tags at all. Those are usually not reported, because the central repository should automatically add special::not-yet-tagged tags to them; however this is sometimes not the case, especially for local packages or packages that have recently been added in the archive, so this option is available to spot such packages.

   Uitoolkit and not interface

   lists all packages which have uitoolkit tags but not interface tags. This might be a normal situation, especially for software libraries, but for normal applications it is probably an anomaly worth checking.

   Uitoolkit and not implemented-in

   lists all packages which have uitoolkit tags but not implemented-in tags. This might be normal for documentation about user interface toolkits, however it is an anomaly for applications and libraries, and it's worth checking as well.

   Missing Role

   Ideally every package should have a kind of role in the distribution. However, there are lots of packages which do not yet have a role::* tag. Since the tags in the Role facet are still under discussion, this function is good to bring up food for ideas.

   Specials

   The options starting with "Specials" allow to see packages whose tag sets are special cases compared to all the other packages with similar tags. How this is computed is a bit complex, but this is proving to be very useful to highlight some corner cases that lead to extra reasoning and improvements of the vocabulary.

3. Click on "Edit/Facet dialog" to activate an experimental new feature of debtags-edit to find even more corner cases.

   This will pop up a dialog with two columns: the Has column and the And not column. Both columns list facets.

   If you click on a facet on the "Has" column, the "And not" column will list all the other facets, together with a number. The number represents how many packages have the "Has" facet, but do not have that "And not" facet. The "And not" facets are sorted by increasing values of this number.

   Sometimes all the "And not" facets have similar numbers, but sometimes there are facets with numbers that are definitely smaller than the others. Those facets are very likely to have something in common with the current facet in the "Has" column, since a lot of packages have tags from both facets.

   For instance, it happens at the time of writing that, when selecting the "Game" facet on the left side, the corresponding "Use" facet reports a count which is one order of magnitude less than the others. This is because of the existence of use::gaming, which is normally found in most, if not all, game::* packages.

   What are then those packages that have game::* but not use::gaming? Let's see them: click on "Use" on the right column, then click ok. That will set game:: && !use:: as the base for the package list, and further filtering will be based on them. You can now see what are those packages, and if you want you can check which of them are installed in your computer.

   With game::*, it turns out that there is a reason: there are games (many of which are game::toys) that are not useful for gaming. Some examples are cappuccino, cowsay, polygen and fortune. Howewer, one could realise that these toys have some other use before gaming.

   Maybe a new "use::something" tag is needed?

   The Facet Dialog has helped finding out.

Using debtags

debtags can be useful for tagging in two ways: both to give you hints on what needs to be worked on, and as a low-level tool you can use if you want to automate tagging tasks.

There are three features of debtags you can use to get hints on what to do:

debtags todo:

debtags todo prints a list of packages you have installed which are tagged with the special::not-yet-tagged tags. No need for further explanations: tag them!

debtags todoreport:

this prints a text report about packages that need some work to do. It includes a range of simple to very complex checks to find out problematic areas of the categorization, and the report that it outputs contains explanations about the reasons behind every group of packages. The report also includes packages that are not installed.

The output of debtags todoreport is periodically processed with rst2html and published at http://debtags.alioth.debian.org/todoreport.html

debtags related:

debtags related shows you what packages are similar (tagging-wise) to a given one. Try running debtags related cdcd: you will see a list of other CD players. You can use the -d switch to increase the distance of the search, that is, to include more packages that are "a bit less similar".

You can use this function to check if the list of related packages is what you would expect: if not, you can check the tags of the missing or extra packages to see if there is anything wrong.

You can also use debtags as a low-level backend if you want to experiment with scripted or automated tagging ideas:

debtags tagsearch and debtags tagshow:

They allow you to search the tag vocabulary and show information about a tag.

debtags tag ls, debtags tag add and debtags tag rm:

These commands are intended to be run from scripts, and they list the tags attached to a package and allow you to add or remove tags to it. Try this:

debtags tag ls debtags
debtags tag add debtags game::toys
debtags tag ls debtags
debtags tag rm debtags game::toys
debtags tag ls debtags

Using tagcolledit

tagcolledit is a generic tool to do mass-editing of tags. It is not specific for Debtags, so it does not know about packages, but compared to debtags-edit it can make it easy to do large-scale operations like renaming of tags.

Let's run tagcolledit:

# Get a version of the tag database including the local patch
debtags cat > /tmp/tags
tagcolledit /tmp/tags

The interface of tagcolledit is unfortunately very rough, just like debtags-edit, and it needs some explanation. It is divided in two panels, like Midnight Commander, and every panel works in the same way: it has a filter, and a list of matching items (that are packages in our case).

Now try doing this: in the left panel, select the uitookit::qt tag. In the right panel, select the implemented-in::c++ and the interface::x11. Now, QT is a C++ library, so one would expect most of the QT applications to have an X11 interface and to be implemented in C++. With tagcolledit it's very easy to have a look at the left panel, select groups of packages that need fixing (the list supports multiple selection), then right click on the selection and choose "Copy to other panel". This will add the tags implemented-in::c++ and interface::x11 to the selected packages, and they will show up in the other panel as well.

You can do many other tricks with tagcolledit: merge or intersect tag sets, add a tag to a group of items or even to all of them (right click on the tag in the filter: you will see the "Add to all" and "Remove from all" functions).

It is however easy to get carried away and make mistakes, such as adding the use::gaming tag to all the games::toys, to find out later that packages such as fortune or cowsay cannot really be used to play games.

Once you are finished with tagcolledit, you can save the tag database and quit. If you are happy with your changes you will want to integrate them in the main tag database: first create a patch:

debtags mkpatch /tmp/tags > /tmp/tags.patch

This will create a tag patch with the changes you have made with tagcolledit. You can then add your patch to the local tag patch:

cat /tmp/tags.patch >> ~/.debtags/patch

And see your changes in all the Debtags-aware applications.

Then you can submit it, with either debtags-edit or debtags submit.

You can even submit it directly:

debtags submit /tmp/tags.patch

Automated tagging

While generally there is a need for a smart brain to do good categorization, there are some tagging tasks that can be automated. For example, it's good enough to assume that all packages that depend on libgtk2.0 will need a uitoolkit::gtk tag, while all packages that depend on libstdc++6 will probably need an implemented-in::c++ tag.

These ideas have brought to creating autodebtag, which is a Perl script that does all kinds of automated reasoning that I and other people in the debtags-devel list conceived so far. Then it prints out a tag patch that can be evaluated, tested and submitted to the central tag database.

autodebtag has also some experimental code to try and bring categorization data from the debram database into the Debtags world.

Benjamin Mesing is also experimenting with setting up a smart Bayesian engine to infer new tags based on what tags are there now. This has very promising and exciting possibilities: there is a prototype available in the autodebtag subversion repository, and it needs more people to play with it.

Adopting a tag or a facet

If you regularly try out all image manipulation software, if you are a Gnome expert, if you are addicted to role-playing games, you might consider adopting a tag.

Adopting a tag means that you take responsibility for checking from time to time that all packages that should have your tag actually have it. It is a very important task, because it introduces a level of reliability and guaranteed accurateness into some parts of the tag database.

For example, Stefano Zacchiroli is an Ocaml guru, and is taking care of the implemented-in::ocaml tag. You will notice that all Ocaml applications have the implemented-in::ocaml tag, and thanks to Stefano you can rely on that tag to be accurate. We need more people to do what he is doing.

To adopt a tag, just send an e-mail to the debtags-devel list, introduce yourself and tell what you want to adopt. If it's already taken, you can team up with the others and share the brainwork.

libdebtags1-dev, python-debtags, libdebtags-perl

If you look inside the code of debtags and debtags-edit, you will notice that most of what they do related to tags is calling functions from the libdebtags1 or libtagcoll1 libraries. libtagcoll1 provides generic manipulation functions for tagged stuff, while libdebtags1 uses libtagcoll1 to implement all Debtags functionality.

If you like programming in C++, install libdebtags1-dev; from there, you can either look at debtags and debtags-edit source code for examples (debtags is particularly straightforward) and use /usr/share/doc/libdebtags1-dev/html/index.html for the doxygen documentation.

The documentation still has missing parts. If you have questions, no matter how silly they could seem, please ask them freely in debtags-devel list: I have promised to myself that I will turn every answer I give into more documentation of libdebtags1-dev.

If you instead like coding in Perl or Python rather than C++, you can install either libdebtags-perl or python-debtags: they are swig-generated bindings to libdebtags1.

Again, for whatever documentation or examples that are missing, ask in debtags-devel, and every answer will also contribute to improve the documentation and examples in the packages.

This is an example Python code that plays with Debtags a bit (please teach me how to use iterators properly):

import Debtags

# Instantiate the simple Debtags class
dt = Debtags.DebtagsSimple(0)

# Get some information from the vocabulary
voc = dt.vocabulary()
print "Facets:"
for a in voc.getFacets().getIterable():
   print "%s: %s %s" % (a.name(), a.field("Status"), a.field("Nature"))

# List the accessibility tags
fac = voc.getFacet("accessibility");
print "Tags in 'accessibility::':"
for a in fac.tags().getIterable():
   print a.name()

# Print the tags of ``debtags``
tags = dt.getTags("debtags")
for a in tags.getIterable():
   print a.name()

# List the image editors in Debian
tags = Debtags.TagSet()
tags.insert(voc.getTag("use::editing"))
tags.insert(voc.getTag("media::rasterimage"))
editors = dt.getPackages(tags)
print "Raster image editors:"
for a in editors.getIterable():
   print a.name()

This is the same example, written in Perl:

#!/usr/bin/perl -w
use strict;
use warnings;
use Debtags;

my $dt = Debtags::DebtagsSimple->new(0);
my $voc = $dt->vocabulary();

print "Facets: ",
  join(', ', map { $_->name() }
    @{$voc->getFacets()->getIterable()}), "\n";

my $fac = $voc->getFacet("accessibility");
print "Tags in 'accessibility::': ",
  join(', ', map { $_->name() }
    @{$fac->tags()->getIterable()}), "\n";

print "Tags for 'debtags': ",
  join(', ', map { $_->fullname() }
    @{$dt->getTags('debtags')->getIterable()}), "\n";

my $tags = Debtags::TagSet->new();
$tags->insert($voc->getTag('use::editing'));
$tags->insert($voc->getTag('media::rasterimage'));
my $editors = $dt->getPackages($tags);
print "Raster image editors: ",
  join(', ', map { $_->name() } @{$editors->getIterable()} ), "\n";

libapt-front

libapt-front is the next-generation wrapper for libapt. The goal of the project is having a library to base all package managers on, and to have it able to access different sources of metadata besides libapt. Including Debtags.

libapt-front is still under heavy development, and Debtags support is still not implemented. However, I have commit access to the libapt-front repository and I've recently been able to take libdebtags1 where I wanted it to be in order to support libapt-front: expect exciting news from this corner of Alioth!

Effective View Navigation

Prof. G. W. Furnas, who usually writes good stuff, made a very nice research (check it out!) and found 4 properties that something should have to be easy to navigate. He called them the properties of Effective View Navigation or EVN properties [FURNAS]:

1. (EVT1) The number of outgoing links must be "small" compared to the size of the structure
2. (EVT2) The distance between pair of nodes in the structure must be "small" compared to the size of the structure
3. (VN1) Every node must contain some valid information (called residue) on how to reach any given other node
4. (VN2) The information associated to each outgoing link (called outlink-info) must be small.

I did some research on this idea related to categorization [TAGCOLL], and designed an algorithm that is able to present navigational choices of tags in a way that is compatible with the 4 EVN requirements. It is a good start, and recent debtags-edit are making use of it in the filter, to reduce the amount of facets when there are too many to display.

This algorithm has potential, but the results are still not perfect: you can now start the navigation with 19 facets instead of the 32 available, but that's still far beyond 7 plus or minus 2. The main cause of this is that tag data is still incomplete: the "TODO Specials" features of debtags-edit are there to expose those packages that are causing problems.

From my experience, however, I see that algorithm as a point of departure rather than a point of arrival, and more smart navigation ideas are waiting to come to life.