Spectrum Wars

25.07.2015 14:42

SpectrumWars will be a programming game where players compete for bandwidth on a limited piece of radio spectrum. It's also apparently a title that tends to draw attention from bored conference attendees and European commissioners alike.

SpectrumWars XKCD spoof.

with apologies to Randall Munroe

With the CREW project nearing completion, it was decided that one of the last things created in the project will be a game. This might seem an odd conclusion to five years of work from 8 European institutions that previously included little in terms of what most would consider entertainment. We built testing environments for experimentation with modern wireless communications aimed at the exclusive club of people that know what CSMA/CA and PCFICH stand for. In the end, I'm not sure we managed to convince them that what we did was useful to their efforts. Much less I guess the broader public and potential students that would like to study this field.

What better to correct this than to present the need for smart radio testbeds in the form of a fun game? The end of the project is also the time when you want to show off your work in way that reaches the broadest possible audience. Everything to raise the possibility that powers-that-be grant you funding for another project.

In any case, SpectrumWars is a fun little thing to work on and I gladly picked it up in the general lethargy that typically suffuses European projects on their home stretch. The idea stems from a combination of the venerable RobotWar game concept, DARPA Spectrum Challenge and some previous demonstrations by the CREW project at various conferences. Instead of writing a program for a robot that fights off other robots, players program radios. Turning the turret becomes tuning the radio to a desired frequency. A spectrum analyzer replaces the radar and the battle arena is a few MHz of vacant spectrum. Whoever manages to be best at transferring data from their transmitter to their receiver in the presence of competition, wins.

A desk-top setup for SpectrumWars with VESNA sensor nodes.

In contrast to the dueling robots, the radios are not simulated: their transmissions happen in the real world and might share the spectrum with Wi-Fi and the broken microwave next door. To keep it fun, things are simplified, of course. The game abstracts the control of the radios to the bare minimum so that a basic player fits into 10 lines of code, and a pretty sophisticated one into 50. This is not the DARPA Challenge where you need a whole team to have a go (unfortunately, there's no five-figure prize fund either).

So far I've made a hardware-agnostic game controller implementation in Python that takes care of all the dirty little details of running the game and defines the player's interface. The radios are VESNA nodes with Texas Instruments CC2500 transceivers and special firmware, but other CREW partners are currently working on supporting their own hardware. A USRP is used as a spectrum sensor that is shared by all users. While a lot of things are still missing (a user-friendly front-end in particular), it was enough to confirm so far that playing with the algorithms in this way is indeed fun (well, at least for my limited idea of fun).

I gave some more details in a talk I had at the Institute last Friday (slides), or you can go check the draft documentation I wrote or the source, if you feel particularly adventurous.

Title slide from the SpectrumWars talk.

In the end, SpectrumWars is supposed to look like a web site where tournaments between algorithms are continuously performed, where one can submit an entry and where a kind of a scoreboard is held. I'm not optimistic that it will be in fact finished to the point where it will be presentable to the general public in the couple of months left for the project. However I think it would be a shame if it went directly into the bit bucket together with many other things when the project funding runs out.

While the future might be uncertain at the moment, I do have a tentative plan to maybe set up an instance of SpectrumWars at the CCC congress in December. The inherently crowded 2.4 GHz spectrum there would certainly present an interesting challenge for the players and the congress does have a history of unusual programming games.

Posted by Tomaž | Categories: Life | Comments »

From F-Spot to Shotwell in Jessie

14.07.2015 20:24

F-Spot photo manager was dropped in Debian Jessie in favor of Shotwell. While Shotwell supports automatically importing data from F-Spot out-of-the box, the import is far from perfect. Here are some assorted notes from migrating my library. Note that this applies specifically to the version in Jessie (0.20.1).

  • Shotwell does not support storing multiple versions of a photo (F-Spot created a new version of a photo for each edit in GIMP, for instance). On import, each version is stored as a separate photo in Shotwell.

  • The tag hierarchy is correctly imported from F-Spot. However sometimes a tag is imported twice: once in its correct position in the hierarchy and once at the top level. This is because F-Spot at some point started embedding tags (without hierarchy data) inside image files themselves in the Xmp.dc.subject field. Shotwell import treats the F-Spot hierarchy and the embedded tags separately, resulting in duplication.

    One suggestion is to remove the embedded tags before import. However, this further modifies files (which shouldn't be modified in the first place, but you can't do anything about that now). Upon removing the field, the exiv2 tool also seems to corrupt vendor-specific fields in JPEG (I don't think truncating the entry warnings are as benign as this thread suggests.

    A better way is to ignore the embedded tags on import. Apply this patch, recompile the Shotwell Debian package and use the modified version to import data from F-Spot:

    --- shotwell-0.20.1.orig/src/photos/PhotoMetadata.vala	2014-03-04 23:54:12.000000000 +0100
    +++ shotwell-0.20.1/src/photos/PhotoMetadata.vala	2015-07-12 13:11:34.021751079 +0200
    @@ -857,7 +857,6 @@
         private static string[] KEYWORD_TAGS = {
    -        "Xmp.dc.subject",

    After importing, you can revert back to the original version, shipped with Debian. The embedded tags are only read once on import.

  • Shotwell doesn't support GIF images. Any GIFs are skipped on import (and noted in the import log) As crazy as that sounds, I had some photos in my library from an old feature phone that saved photographs as GIF files. I converted them to PNG and manually re-imported.

  • Time and date information is not imported from F-Spot at all. Shotwell reads only the data from the EXIF JPEG header. If you adjusted creation time in F-Spot, those modifications will be lost.

    Unfortunately, F-Spot itself seems to have had problems with storing this data in its database. It looks like the timestamps have been stored in different formats over time without proper migrations between versions. Looking at my F-Spot library, various photos have +1, 0 and -1 hour offsets compared what they probably should be.

    I gave up on trying to correct the mess in the F-Spot library. I made a Python script that copied timestamps directly from the F-Spot SQLite database to the Shotwell database, but only when the difference was more than 1 hour. The script is quite hacky and probably too dangerous for general consumption. If you have similar issues and are not afraid of SQL, send me a mail and I'll share it.

In the end, Shotwell feels much more minimalist than F-Spot. Its user interface has its own set of quirks and leaves a lot to be desired, particularly regarding tag management. On the other hand, I have yet to see it crash, which was a common occurrence with F-Spot.

Posted by Tomaž | Categories: Code | Comments »

Python PublicSuffix release 1.1.0

10.07.2015 16:46

The publicsuffix Python module is one of the most widely used pieces of open source software that resulted from my years at Zemanta. With around 1000 daily downloads from the package index, it's probably only second to my Python port of Unidecode. I've retained the ownership of these packages even though I moved on to other things. Being mature pieces of specialized code, they are pretty low-maintenance, with typically one minor release per year and an occasional email conversation.

The publicsuffix 1.1.0 release had some interesting reasons behind it, and I think it's worth discussing them a bit here.

Since the start, publicsuffix shipped with a copy of Mozilla's Public Suffix List as part of the Python package. My initial intention was to make the module as easy to use as possible. As the example in the README showed, if you constructed the PublicSuffixList object without any arguments, it just worked, using the built-in data. The object supported loading arbitrary files, but this was only mentioned in the docstring. My reasoning at the time was that Public Suffix List was relatively slow-moving. For quick hacks it wouldn't matter if you used slightly out-dated data and serious users could pass in the exact version of the list they needed.

Later I started getting reports that other software started using publicsuffix with the assumption that the built-in data is always up-to-date. A recent example reported by Philippe Ombredanne was url-py. Some blame is definitely with authors of such software not reading the documentation. My README did have a warning about the default list being possibly out-dated, although that fact was probably not stressed enough. However, after reading complaints over time, I felt that perhaps publicsuffix could offer an interface that was less prone to errors like that.

I looked through the possibilities on how to improve things by looking at the quite numerous publicsuffix forks on GitHub and Public Suffix implementations in other languages.

The publicsuffix package with a built-in copy could be updated each time Mozilla updates the List. This would make the package decidedly less low-maintenance for me. It would also require that every software that uses publicsuffix regularly re-installs its dependencies and restarts to refresh the data used by PublicSuffix. From my experience with deploying server-side Python software, this is quite an unreasonable requirement.

Another approach would be to fetch the list automatically from Mozilla's servers at run-time. However, I don't like doing HTTP requests without this being explicitly visible to the developer. Such things have led to unintentional storms of traffic to unsuspecting sites. Also, for some uses, the latest version of the list from Mozilla's servers might not even be what they need (consider someone processing historical data).

Finally, I could simply say it's the user's responsibility to supply the algorithm with data and not provide any default. While this requires a bit more code, it also hopefully gives the developer a pause to think what version of the List is relevant to them.

The latter is the approach I went with in the end. I deprecated the argument-less PublicSuffixList constructor with a DeprecationWarning. I also added a convenience fetch() function that downloads the latest List from the web and the result of which can be directly passed to the constructor. This makes the request explicit (and explicit is better than implicit is, after all, in the Zen of Python) and only little less convenient. I opted against implementing any kind of caching in publicsuffix since I think that is something specific to each application.

In conclusion, you can't force people to carefully read the documentation. People will blame you for bugs in other people's software (incidentally, this release also works around a packaging problem due to a long-standing bug in Wheel). It's worth thinking about defaults and carefully picking what examples you show in the README.

Posted by Tomaž | Categories: Code | Comments »