vesna-drivers git visualization

16.06.2014 17:41

Further in the context of two years in development of firmware for VESNA sensor nodes in Logatec, here's another view on the firmware. I used git-big-picture to graph the branching and merging of the source code.

We use a private GitHub repository to collaborate on the code. At the time of writing, it had 16 forks and 7 contributors to the master branch. Before that we used Subversion, but switched to git soon after I joined the team.

vesna-drivers repository branching visualization

Red are commits pointed to by tags, blue are branch heads and white are merge and bifurcation commits. Other uninteresting commits are not shown - zero or more intermediary commits are implied by graph edges. Time, or rather entropy, roughly increases from left to right.

Below is a detail from that picture. If you're curious, you can also download the complete graph as a rather large PDF (most branch names and tags have been redacted though).

vesna-drivers repository branching visualization, detail

At the first glance, this looks pretty chaotic. Definitely our academic environment is a bit less strict regarding the code structure than what you might see in a production project. That's not necessarily a bad thing. Many people here are researchers first and software developers second. For many, this has been their first encounter with source version control.

On the other hand, the whole point of this repository is that work can be shared. Pieces that have been useful in one project are often useful again in another. Searching 16 repositories and countless branches for a driver you might reuse isn't very fun. So some kind of structure is a must.

It's hard to get this balance right tough. On one hand you don't want to be too strict when accepting code to the git master, since then you have less code to share. Often it's even impossible for the reviewer to run-test the code since he might not have the hardware necessary. On the other hand, merging code that will be a maintenance hell later on is counter productive as well. There's no use trying to share code that will cost people more time to debug it than it would to rewrite it from scratch.

We're currently trying to go with a system of GitHub pull requests and a Wiki page that lists all the projects in private branches and I think setting up automated builds was worth the effort. But I guess after two years we're still trying to find the sweet spot.

Posted by Tomaž | Categories: Code | Comments »

Evolution of VESNA firmware size

11.06.2014 21:23

Yesterday I got an idea to plot how the size of the firmware images for VESNA sensor nodes evolved over time. Thanks to my diligent tagging of releases in git and making sure binaries can be built with a single make, all it took was a short bash script to come up with this:

Size of binary firmware image over time.

Only now do I realize that we've been polishing this code base for two years now.

When we originally started working on firmware that would run on spectrum sensing nodes in the Logatec testbed, a decision was made to develop everything from scratch and not go with an existing operating system like Contiki. The rationale was that we don't need all of its facilities and making something from scratch will be simpler than learning and building on existing work.

As it usually happens in such cases, over time we basically developed our own, small operating system. Given all the invested time, it is now hard to make a break from it, even when we have applications that would benefit from a standard platform like Contiki.

Looking at the graph, I'm actually surprised that the size of the firmware didn't increase more than it has. Keep in mind that these binary images are uploaded over a flaky ZigBit network where the best throughput is in the hundreds-of-bytes-per-second range. From the upload times to 50 nodes it certainly felt like it has increased a lot.

I didn't look into what features caused the most size increase. I'm pretty sure the recent large jump between versions 2.40 and 2.42 was because of a new SPI and microSD card driver. Also, one of the size decreases early on was after we added -ffunction-sections and -fdata-sections options to the GCC.

Posted by Tomaž | Categories: Code | Comments »

Concentrated Python cuteness

20.03.2014 19:45

Yesterday, I wrote the following piece of Python code to answer a question asked on IRC by Jure. It was more of a joke than a serious suggestion.

def list_pages(page_num, cur_page, n):
	k = sorted([	0, 2*n+1, 
			cur_page-(n+1), cur_page+n, 
			page_num-(2*n+1), page_num])
	return range(1, page_num+1)[k[1]:k[-2]]

What it does is take a list of page numbers from a total of page_num pages. If possible, the list is centered on cur_page and includes n pages in front and back. At the edges, it still returns the list of the same length. For example:

list_pages(10, 1, 2)  = [1, 2, 3, 4, 5]
list_pages(10, 2, 2)  = [1, 2, 3, 4, 5]
list_pages(10, 3, 2)  = [1, 2, 3, 4, 5]
list_pages(10, 4, 2)  = [2, 3, 4, 5, 6]
list_pages(10, 5, 2)  = [3, 4, 5, 6, 7]
list_pages(10, 6, 2)  = [4, 5, 6, 7, 8]
list_pages(10, 7, 2)  = [5, 6, 7, 8, 9]
list_pages(10, 8, 2)  = [6, 7, 8, 9, 10]
list_pages(10, 9, 2)  = [6, 7, 8, 9, 10]
list_pages(10, 10, 2) = [6, 7, 8, 9, 10]

This is likely one of the most incomprehensible pieces of Python code I have ever written. The idea came from Python's cutest clamp function. It's a kind of cleverness I never want to see in a serious product.

The version Jure ended up using took around 30 lines of code to accomplish the same thing. However even with descriptive variable names it's not immediately obvious to me what it does or how it does it. If I would come across it without any additional comments, it would probably take some time and a few tests to see what is its purpose.

I failed to come up with a version that would be self-explanatory.

Perhaps comprehensive code documentation is dead these days, but I think this is one example where an English sentence can be much more expressive than code.

Update: In case anyone is seriously considering using Python code above, please be warned that it does not work for a general case.

Posted by Tomaž | Categories: Code | Comments »

Moving a SSL certificate to a hardware token

05.01.2014 21:35

This is how you move a client side SSL certificate from Firefox to a hardware cryptographic token. It's something I do just rarely enough so that I have to look it up every time. And due to the chaotic nature of OpenSC documentation it's not easy to find all the steps. Here is a quick guide for future reference:

This assumes that OpenSC is already installed on the system and working correctly. I'm using a Schlumberger Cryptoflex USB token.

Cryptoflex works with 2048 bit RSA keys. I haven't tried larger ones.

First export the private key and certificate to a PKCS #12 file: Edit → Preferences → Advanced → Certificates → View Certificates → Your Certificates → Backup.

You can verify that it worked by:

$ openssl pkcs12 -in file.p12

Now insert the USB token or a smart card into the reader. You can inspect existing contents by:

$ pkcs15-tool -D

The Cryptoflex 32K doesn't seem to have enough memory for two key pairs, so you have to delete any existing content before uploading a new certificate. It might be possible to just delete individual files from the token, but I couldn't figure it out, so I just erase the whole device and setup everything from scratch.

First erase the token and setup the PKCS #15 structure on it. The default transport key offered by OpenSC works.

$ pkcs15-init --erase-card
$ pkcs15-init --create-pkcs15

Create a PIN and PUK entries on the token:

$ pkcs15-init --store-pin --auth-id 1 --label "My PIN"

Now upload the key you exported from Firefox to the token and protect it with the PIN you entered previously:

$ pkcs15-init -S file.p12 -f PKCS12 --auth-id 1

Verify that it has been written to the token correctly using pkcs15-tool -D. You can now remove the certificate from Firefox' software storage. You can do that from the certificate manager. You have to remove the token from the system first, because the Firefox' UI hides certificates in software storage if a hardware token is present.

Make sure you keep a safe backup of the file.p12 (and remember the passphrase). It should be impossible to retrieve the private key back from the hardware token so this is now your only way to recover it in case you want to move it to a new device in the future.

Some more background info is available on the old OpenSC wiki. It's not linked from anywhere right now because supposedly they have a new wiki, but a lot of documentation didn't make it there yet.

Posted by Tomaž | Categories: Code | Comments »

OpenSC on Wheezy

27.10.2013 11:43

One of the things that broke for me on upgrade to Debian Wheezy was smartcard support in Iceweasel. I regularly use a Schlumberger Cryptoflex USB key to authenticate on websites using client-side SSL certificates, so fixing this was kind of important to me.

OpenSC documentation is a mess and from terse error messages it was hard to make heads or tails of what was actually broken. So here's what I had to do make authentication work again in the browser.

First, fixing the most obvious thing: with the introduction of multiarch the PKCS #11 module has moved from /usr/lib/opensc-pkcs11.so to /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so. This means you have to correct the path in Iceweasel. Go to Preferences, Advanced, Certificates, Security Devices and select the OpenSC module there. Click Unload to remove the module and then Load to load the module from the new path.

Also, you might have noticed that mozilla-opensc package was removed in Wheezy. I'm not sure if it was even required in the previous release, but it's definitely not needed now.

Second, the version of OpenSC shipped with Wheezy only supports accessing the smartcard readers through the pcscd daemon. You have to install the pcscd package or OpenSC will not detect any readers.

$ opensc-tool -l
# Detected readers (pcsc)
Nr.  Card  Features  Name
0    Yes             Axalto/Schlumberger/Gemalo egate token 00 00

Now for the tricky part. With the changes above, I still got a very helpful error message whenever I tried connecting to a secure website: A PKCS #11 module returned CKR_GENERAL_ERROR, indicating that an unrecoverable error has occurred. (Error code: sec_error_pkcs11_general_error).

sec_error_pkcs11_general_error message in Iceweasel

Running a test with the pkcs11-tool showed that there was something wrong with the signing operation:

$ OPENSC_DEBUG=9 pkcs11-tool --module /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -t -l
Using slot 1 with a present token (0x1)
Logging in to "OpenSC Card (tomaz)".
Please enter User PIN: 
C_SeedRandom() and C_GenerateRandom():
  seeding (C_SeedRandom) not supported
  seems to be OK
Digests:
  all 4 digest functions seem to work
  MD5: OK
  SHA-1: OK
  RIPEMD160: OK
Signatures (currently only RSA signatures)
  testing key 0 (Private Key) 
... lots of debug output skipped ...
iso7816.c:103:iso7816_check_sw: Command incompatible with file structure
card-flex.c:1067:cryptoflex_compute_signature: Card returned error: -1200 (Card command failed)
sec.c:56:sc_compute_signature: returning with: -1200 (Card command failed)
card.c:330:sc_unlock: called
pkcs15-sec.c:380:sc_pkcs15_compute_signature: sc_compute_signature() failed: -1200 (Card command failed)
card.c:330:sc_unlock: called
reader-pcsc.c:548:pcsc_unlock: called
framework-pkcs15.c:2721:pkcs15_prkey_sign: Sign complete. Result -1200.
misc.c:59:sc_to_cryptoki_error_common: libopensc return value: -1200 (Card command failed)
pkcs11-object.c:691:C_SignFinal: C_SignFinal() = CKR_GENERAL_ERROR
error: PKCS11 function C_SignFinal failed: rv = CKR_GENERAL_ERROR (0x5)

Aborting.

This seems to be a bug in the 0.12.2-3 version of the opensc package. Luckily, this is fixed in 0.13.0-3 that is currently in Unstable. Upgrading is pretty trivial and doesn't depend on upgrading a lot of other packages on the system.

With this upgrade in place, everything works again for me as it did in Squeeze.

Update: You might want to also upgrade libpcsclite1 and pcscd to versions from Unstable (1.8.10-1). With versions from Wheezy I'm still occasionally getting errors.

Posted by Tomaž | Categories: Code | Comments »

time_t on embedded systems

03.10.2013 18:44

Recently I got pointed to this slide deck from Theo de Raadt on moving to a 64-bit type for representing time in OpenBSD. It addresses ways of overcoming the limitation of the way Unix-based systems represent time. Among other things it also mentions the proliferation of the POSIX standard into the domain of embedded systems and the problematic combination of the timer overflow in the not-so-far future and the particularly long life cycles of such systems.

One interesting observation he makes is that applications often do arithmetic on the clock (for instance when calculating timeouts) and that might produce bugs even before the clock itself overflows. So things will probably start breaking well before the January 2038 deadline.

Regarding embedded systems I would like to add that this problem most likely affects a much larger set than those running POSIX-like operating systems.

One reason is that most systems that are programmed in C use the standard C library. While the C standard itself doesn't dictate how time is stored, all of the most common library implementations come from the POSIX world and bring in its definition of time_t as a signed 32-bit counter for the number of seconds from 1 January 1970. For instance, software on VESNA uses newlib. Its clock used to count seconds from 1 January 2012, but that was later changed to the standard POSIX time for the sake of simplicity of using built-in time-handling functions in the C library.

The other reason is that embedded systems often rely on hardware counters to keep the clock. ARM microcontrollers from STM for example have a 32-bit counter in the RTC peripheral. As far as I know it is a true 32-bit counter, meaning it has one more usable digit than the signed 32-bit time_t in POSIX (which adds another 68 years to its lifetime). However, there is no simple way around this limitation in software. You could start using a software counter instead, but that can potentially change response times to other interrupts that might be critical on a microcontroller system. I don't want to even imagine retrofitting that to a system that hasn't been touched in years and might control something big and expensive.

Anyway, for the moment I will go with the prevalent belief that code running on VESNA won't be around for that long and plan to book a holiday in the southern hemisphere for 2038.

Posted by Tomaž | Categories: Code | Comments »

Python Unidecode release 0.04.14

21.09.2013 14:40

Yesterday I released Unidecode 0.04.14, a new version of my Python port of Sean Burke's Text::Unidecode Perl module for transliterating Unicode strings to 7-bit ASCII.

Together with a few other minor changes, this release reverts one quite controversial change in the previous version. In the new version, Latin characters with diaeresis (ä, ö, ü) are again simply stripped of accents instead of using German transliterations (ae, oe, ue).

Using German transliterations instead of simple language-neutral accent stripping was the most often requested change to Unidecode. However, after that change was released, the most often reported bug was again concerning the transliteration of these characters. This reaction was interesting and several lessons have been learned from it.


Before diving into the problem of transliteration at all, it was obvious that people wrote code under the assumption that Unidecode will not change the transliterations over time. Apparently the 0.04.13 release broke many websites because of this, since the most popular use of Unidecode is to generate ASCII-only URLs from article titles. This is interesting because each and every release of Python Unidecode contained changes to the mappings between Unicode and ASCII characters, as it was clearly stated in the ChangeLog. This bug only became apparent once transliteration of an often used character, like ä, was changed.

So, the lesson here is that if you are using Unidecode for generating article slugs, you should only do transliteration once and store it in the database.


Now, the issue with automatic transliteration is that it's hard to do it right. In fact, it's something that requires understanding of natural language, itself a strong AI problem. If you want to get a rough idea what it involves and what kind of trade-offs Unidecode does, I suggest reading Sean Burke's article about it. Here's a relevant quote from it:

The grand lesson here is tht if y lv lttrs ot, ppl cn stll make sense of it, but ifa yaou gao araounada inasaeratainaga laetataerasa, the result is pretty confusing.

Above explains nicely why having German transliterations in Unidecode doesn't work. While it might be that German is the most common use of these characters, it makes transliteration for other languages using the same characters much worse. As Sean points out, it is much easier for a German speaker to recognize words with missing es than it is for instance for a Finnish reader to deal with extra es (compare Hyvää päivää with Hyvaeae paeivaeae or Wörterbuch with Worterbuch). It was my error that I did not remember this argument before accepting the German transliteration patch.

However, this is such a popular issue that even Wordpress implements the German transliteration as the only language-specific exception in their own transliteration tables. It should also be pointed out though that this simple fix does not actually make German transliteration perfect. There is an issue with capitalization (without understanding the context you can't know whether to replace upper-case Ä with Ae or AE).

The solution here, which I will suggest in the future to anyone that will report this as a bug, is that you should use a language-specific transliteration step before using Unidecode. You can find several of them on the web. Some, like Unihandecode, have been built on top of Unidecode.

One thing you should be aware though, is that these language-specific solutions can give a false sense of correctness. You are now relying on responsible people setting the string language correctly, when many don't even get the string encodings right. Also, can you be sure that all of the input will be in the language that was specified? An occasional foreign visitor might be much more upset with a wrong language-specific transliteration of her name than a somewhat language-neutral one provided by Unidecode.

In any case, you should be aware that using automatic transliteration to produce strings that are visible to general public will lead to such problems. This is something that developers of Plone and Launchpad got to experience first hand (although I believe the latter was not due to Unidecode).


In conclusion, I will now be much more careful accepting patches to Unidecode that deal with language-specific characters. In contrast to Sean I don't have a Master's degree in linguistics and only have a working knowledge of three languages. This makes me mostly unqualified to judge whether proposed changes make sense. Even if submitters put forth good arguments, they probably don't have a complete picture of which other languages their change might affect. Even though they didn't raise as much dust as this most recent one, I'm now actually afraid of how much damage was caused by those other few changes to Sean's original transliteration tables I accepted.

Posted by Tomaž | Categories: Code | Comments »

Hot plugging monitors in XFCE

17.09.2013 13:59

Ever since Debian Wheezy has been released, I've been putting off the upgrade of my non-server machines because I don't want to switch from my cozy GNOME 2 desktop. As kind of a test trial, I've been using Wheezy with XFCE on my work laptop since mid-July. With some work I managed to get it pretty close to GNOME 2 functionality I am used to. However it still feels like a significant step backwards in usability and it has several annoying features that I was so far unable to work around.

I might post a list of tweaks I did later on in another post. For now, here is a somewhat ugly hack to work around the fact that XFCE never fails to do exactly the wrong thing when a monitor has been added or removed from the system. As I often move between a docking station, external monitor and an occasional projector, it's extremely annoying to have to drop into a terminal all the time to poke around xrandr.

This is based on this post I found. The original solution didn't work for me and was a bit more convoluted than it strictly needed to be. Plus having some error checking is always nice.

Any way, put the following into /etc/udev/rules.d/20-monitor.rules to tell udev to call a shell script on a monitor hot plug event:

ACTION=="change", SUBSYSTEM=="drm", ENV{HOTPLUG}=="1", ENV{DEVNAME}=="dri/card0", RUN+="/etc/udev/monitors.sh"

The corresponding script in my case looks like this:

#!/bin/bash

set -e

XUSER=`w -h -s|awk '$3~"^:0"{print $1; exit 0}'`
if [ -z "$XUSER" ]; then
	echo "No one logged in, exiting."
	exit 0;
fi

XHOMEDIR=`getent passwd $XUSER|cut -d: -f6`

export DISPLAY=":0"
export XAUTHORITY="$XHOMEDIR/.Xauthority"

# Adjust the following to fit your needs

function dock {
	xrandr --output DisplayPort-2 --left-of LVDS --primary --auto
}

function undock {
	xrandr --auto
}

if xrandr|grep -q 'DisplayPort-2 connected'; then
	dock
else
	undock
fi

By the way, to debug scripts called from udev, use:

# udevadm control --log-priority=debug

This causes udev to log standard output and error streams emitted by scripts to syslog and is pretty useful when a script mysteriously doesn't seem to do what is supposed to.

The original warnings still apply. This solution only works when there is one graphical log in. It also doesn't work when you first turn on the computer or when the configuration of monitors change while the laptop was suspended.

For these cases I have a shortcut to monitors.sh on my desktop which I can click in anger when my high-DPI 24" flat panel starts pretending it has resolution of a Game Boy LCD.

Posted by Tomaž | Categories: Code | Comments »

Unbreaking GNU Radio GUI

01.09.2013 21:14

Here's a recipe how to solve a particular problem with GNU Radio. More or less for my own record as I'm pretty sure I'll come across this same problem again on some other machine and it took some time to figure it out.

If suddenly GNU Radio instrumentation using the WX GUI (like FFT and waterfall plots) look all strange and broken like this:

GNU Radio WX GUI without OpenGL

the cause of the problem is that Python OpenGL bindings are not working for some reason. In this case GNU Radio will automatically fall back onto somewhat simpler widgets without any warnings in the console. These backup widgets however seem to be very outdated or downright broken. A lot of buttons and controls are missing and the waterfall plot is completely useless.

Package that needs to be installed on Debian is python-opengl.

After this is corrected, the same WX GUI FFT Sink as above will be shown like this:

GNU Radio WX GUI with OpenGL

You can override auto-detection of a working OpenGL installation by modifying the following part of etc/gnuradio/conf.d/gr-wxgui.conf. This is useful because if you force the style to gl you will actually see the error that is causing the code to fall back to non-GL widgets:

[wxgui]
# 'gl', 'nongl', or 'auto'
style = auto
Posted by Tomaž | Categories: Code | Comments »

CuteCom fixes

19.08.2013 16:49

When developing embedded software, having a working serial terminal emulator is almost as important as having a working C compiler. So when working with VESNA I more or less have at least one instance of CuteCom running on my desktop at all times.

While CuteCom does its thing relatively well, the last version released by the original author (0.22.0) has a few minor annoyances that only get bigger with daily use:

  • Program occasionally stops saving terminal settings,
  • device drop-down history starts accumulating empty lines and
  • input widgets are not disabled when serial port is opened in read-only mode.

Fortunately, all of these can be fixed with more or less trivial patches. Unfortunately, there has been zero response to these patches from the upstream author. I have also tried to reach the author of the Debian package via e-mail to ask him if he might include these patches in the version shipped by Debian. It has been now more than a year since these attempts at communication and I have yet to receive any response.

So basically, now I'm just throwing the whole thing into the wind and hoping that maybe, just maybe, at one point someone important will pick it up and I won't have to keep installing a locally-compiled version of this package on every machine I use.

You can get the Debian source package and amd64 binary for Wheezy. If you want to include these fixes in some other distribution of CuteCom (gasp!) I have also been kind enough to nicely package them separately for you.

Posted by Tomaž | Categories: Code | Comments »

Beware of long lines in git

14.08.2013 15:45

These days I use git almost exclusively for version management. Not just for code, but also sometimes for plain text or LaTeX documents. One thing that soon becomes apparent when using git for prose is that default command line tools are very bad at handling long lines. There are some workarounds that improve the situation somewhat, but you have to remember to apply them on a per-repository basis and don't solve all problems.

However, recently I noticed that this deficiency can also become a problem when handling code contributions and pull requests on GitHub.

Take for instance this example where you are merging in some changes from someone else's branch. git diff doesn't show any indication that something may be hidden beyond the right edge of the terminal:

git diff --cached output for modified hello-world.c

A pull request on GitHub fares marginally better. Here you at least get a scroll bar at the bottom that hints at the fact that something may be amiss. However, if this branch would contain some more changes, the scroll bar could be several screen heights below the line that is causing it. Unless you have a mouse that can do horizontal scrolls it becomes practically impossible to review code like that (the view port is also conveniently set to fixed width, so resizing the browser window doesn't help).

To be honest, until yesterday I didn't even notice that you get a scroll bar.

GitHub pull request example.

Only opening the modified file in an editor that will automatically break long lines (and not all of them do that!) will unambiguously reveal the complete content of the pull request.

Modified hello-world.c opened in vi.

So I guess the conclusion would be to not accept pull requests from untrusted sources based solely on git diff output and be mindful of scroll bars on GitHub.

Posted by Tomaž | Categories: Code | Comments »

Playing the Smile song on VESNA

26.06.2013 17:53

A while back I was writing about wireless microphone simulation with VESNA. The reason back then was to be able to use remotely-accessible sensor nodes as targets for spectrum sensing experiments. However when I was writing the direct digital synthesis code it occurred to me that transmitting just a stationary tone is slightly boring. Basically nothing was preventing me from transmitting something more fun.

Since playing back recorded audio has somewhat prohibitive storage requirements for a low-powered sensor node I decided to try with a software music synthesizer. If microcomputers in the 80's were able to do it, there should be no reason why a modern ARM CPU couldn't. I've never actually written a synthesizer before and as usual when playing with sound or graphics experimenting with this code was incredibly exciting.

In the end I implemented a 6-channel wavetable synthesizer. Since now I was more concerned with audio quality than correct spectral envelope I replaced the 2-bit quantization with 1-bit and added delta-sigma modulation. This way I was able to exploit the high bit-rate supported by the CC1101 transceiver to get 4 effective bits of audio signal resolution since delta-sigma moved some quantization noise above audible frequencies. The best (subjective I guess) compromise between resolution and sample rate was at 25 kHz audio sample rate and 16x oversampling.

The most complicated thing in all of this was actually figuring out how to interpret time information in a MIDI file. There's a lot of information about this floating around but in the end understanding how to convert all of those weird music units into seconds and milliseconds took way more time than I thought.

VESNA audio synthesis demo block diagram.

It turns out that the 72 MHz Cortex M3 CPU on VESNA runs such a synthesizer without a breaking a sweat. The limiting factor was interrupt load because the streaming of data to CC1101 had to be done with software bit-banging. Since this use case wasn't exactly planned for VESNA no integrated serial bus peripherals are connected to GPIO lines that are used by CC1101 for data streaming. The software implementation handles at most 400 kbps while the transceiver goes up to 800 kbps, which would allow for one additional bit of audio signal resolution.

At high bit rates the size of the synthesis buffer also starts getting problematic. Since you want the buffer to cover at least a few periods of the lowest audio frequency you quickly run out of 96 kB of RAM on VESNA if the baseband rate is significantly higher than the audio rate.

Anyway, hacking on this has been seriously enjoyable. It was one of those side projects that give you strange looks at the office and are hard to explain to someone without giving a fair amount of background information. But in the end there's just something fundamentally fun about having at your fingertips the power to convert 50 remote devices into wireless microphones transmitting a happy song sung by a cartoon pink pony (because seriously, what else would you want to transmit over a city-wide sensor network?)

You can see a short video demonstration above which should give you an idea of what kind of audio quality is possible with this setup.

Audio synthesis source code is hosted on GitHub if you are curious about the details. Also, if you'll be in Köln two weeks from now, look for my lightning talk about this hack at SIGINT 2013.

Posted by Tomaž | Categories: Code | Comments »

Cookie law compliance

02.06.2013 19:06

On 15 June a new Slovenian law on electronic communications comes into effect. Among other things it implements the European cookie directive which makes Slovenia one of the last countries in the Union to comply with it.

Slovenian law makers decided on a stricter interpretation of the directive than most other countries, making illegal for instance storing any state on a visitor's browsers without her explicit consent. There are very few exceptions to this rule, for example where local state is the only way to implement some functionality (shopping carts for instance) or where this is expected (when you log in with a user name). But otherwise our Information commissioner is quite clear that various tracking cookies and such for anonymous users must go away unless a click-through warning is added to a web page. It remains to be seen to what extent this will be enforced though, especially since parts of the law also attempt to restrict what kind of processing you can do on plain web server access logs.

Since I started writing this blog I've been quite careful to respect the privacy of visitors on my web site. I never used cookies and for most of its existence these pages didn't include any third-party Javascript. I never used Google Analytics or other third-part analytics services and my own web server logs are only used for occasional local statistical processing when I'm curious about the number of visitors to particular articles.

I was therefore somewhat surprised when I was discussing this topic in our local Open data community and we ran some automated tests against my pages. It turns out the situation was not as rosy as I thought.

First problem was that against all odds cookies were getting set for my domain. I tracked it down to jsMath, which I use to typeset mathematical notation in some blog posts. jsMath uses a cookie to store settings you can change using a small toolbox that appears in the lower right corner of the website when a text with mathematical symbols is displayed (I'm quite sure nobody noticed it though).

That cookie isn't problematic however, since changing settings is an explicit action that is expected to save some state (the box also has an option for how long you wish the settings to be retained making that fact even clearer). However for some reason jsMath will always set a default cookie on the first visit, even if you don't touch any settings. That's not OK even though the cookie doesn't include any unique identifiers (and in fact is used solely on the client side, even though the browser will send it to my server on each HTTP request).

I'm not sure whether this is a bug in jsMath or if this is intentional. Anyway, the simple one-line patch below corrects this behavior and retains the setting-saving functionality if you happen to use it:

--- a/uncompressed/jsMath.js
+++ b/uncompressed/jsMath.js
@@ -1834,7 +1834,7 @@ jsMath.Controls = {
   cookie: {
     scale: 100,
     font: 'tex', autofont: 1, scaleImg: 0, alpha: 1,
-    warn: 1, fonts: '/', printwarn: 1, stayhires: 0,
+    warn: 0, fonts: '/', printwarn: 1, stayhires: 0,
     button: 1, progress: 1, asynch: 0, blank: 0,
     print: 0, keep: '0D', global: 'auto', hiddenGlobal: 1
   },

The second problem I had was that a few of my posts embed YouTube videos. That's a problem since YouTube player will drop two Flash Local Shared Objects on the visitor's computer as soon as it is loaded (even if you use the nocookie domain).

To my knowledge it is now impossible to embed a YouTube video on a web site and comply with the Slovenian law unless you provide a click-through warning. Since I find those obnoxious I chose to remove all embedded videos and replace them with static thumbnails that you can click-through to watch the video on the YouTube web page itself.

The other option would be to find some other video hosting service that would not set cookies (if it even exists) or host video files myself (which didn't end well a while ago). Both of these require more time than I'm willing to spend fixing this issue at the moment.

Posted by Tomaž | Categories: Code | Comments »

AFG-2005 follow up

26.05.2013 12:56

Here is some additional information about the GW Instek AFG-2005 function generator I reviewed earlier.

My first actual use of this instrument left a pretty bad impression. My father and I attempted to perform some tests on his home made short-wave transceiver and it failed badly. For some reason most commands we sent from the computer over the USB would either crash the AFG-2005 or produce very unpredictable results so we were not able to do anything useful with it. I'm still not sure what was the reason.

We used a somewhat longer USB cable (maybe 5 m) so one possibility was that data on the cable got corrupted. Considering that there was some high-powered RF hardware near by, I think that is not that far fetched. Linux kernel didn't show any complaints though which is weird. The other theory is that there was some incompatibility with the kernel that was running on the Ubuntu machine we were using. User space code was exactly the same that later ran fine at my place with AFG-2005 connected to my own computer.

Anyway, I published some Python code for controlling the AFG-2005 from a Linux computer. Currently it's pretty rudimentary, but takes care nicely of the specifics of the IEEE-488 implementation on this instrument.

To get it, run:

$ git clone https://github.com/avian2/gwinstek-tools.git

Example code to generate a 100 Hz square wave:

from gwinstek.afg import AFG2000

afg = AFG2000("/dev/ttyACM0")
afg.set_waveform(100, [-1.0, 1.0])

Some more notes regarding things I said in the previous post:

About synchronization issues with USBTMC: Apparently IEEE-488 specifies some special commands to deal with command concurrency and synchronization. This somewhat dated GPIB programming tutorial explains it nicely. AFG-2005 doesn't seem to support neither *WAI nor *OPC commands though.

Regarding error reporting, IEEE-488 specifies an error queue where errors are accumulated and can be read by the instrument controller using the SYSTEM:ERROR? query or cleared using the *CLS command. While the manual doesn't mention it explicitly, AFG-2005 does support this functionality. Python modules I published above use it to raise Python exceptions on command errors.

Posted by Tomaž | Categories: Code | Comments »

Custom display resolutions in QEMU

04.05.2013 19:19

QEMU emulates a VESA-compliant VGA card that supports a static set of standard and a-bit-less-standard display resolutions. However, this set is far from universal and for example doesn't include the wide-screen 1600x900 resolution on my laptop. This means that running a virtualized OS full-screen would always stretch the display somewhat and give a blurry image.

Here's how I fixed that. Following instructions should pretty much work for adding an arbitrary resolution to QEMU's virtualized graphics card (but only when using -vga std). They are based on this patch.

Fetch QEMU source (I used 1.4.0) and edit roms/vgabios/vbetables-gen.c. For instance, to add 1600x900 I applied this patch:

--- qemu-1.4.0.orig/roms/vgabios/vbetables-gen.c	2013-02-16 00:09:22.000000000 +0100
+++ qemu-1.4.0/roms/vgabios/vbetables-gen.c	2013-05-04 11:46:55.000000000 +0200
@@ -76,6 +76,9 @@
 { 2560, 1600, 16                     , 0x18a},
 { 2560, 1600, 24                     , 0x18b},
 { 2560, 1600, 32                     , 0x18c},
+{ 1600,  900, 16                     , 0x18d},
+{ 1600,  900, 24                     , 0x18e},
+{ 1600,  900, 32                     , 0x18f},
 { 0, },
 };

Now re-build the VGA BIOS binary image (apt-get install bcc first):

$ cd roms/vgabios
$ make stdvga-bios

QEMU's make install will not install the image you just built. Instead, it will use an already built binary they ship with the source. Therefore you have to install it manually:

$ cp VGABIOS-lgpl-latest.stdvga.bin $PREFIX/share/qemu/vgabios-stdvga.bin

Of course, replace $PREFIX with the prefix you used when installing QEMU (/usr/local by default). Now when you start a virtualized OS you should see the resolution you added to vbetables-gen.c (for example as reported by xrandr or under Screen resolution in Windows).

Posted by Tomaž | Categories: Code | Comments »