Avian’s Blog

My trusty old Happy Hacking Keyboard has been working pretty reliably for the last four years. After fixing a botched plastic mold and strategically placing a piece of cardboard in its innards that is. Regarding the typing feel it is still my favorite keyboard that doesn't take a lot of space on a crowded desk and I only switch to a regular-sized Logitech when I'm working with EDA programs where I need functions keys a lot.

So I was pretty disappointed when it stopped working a week ago. Checking the kernel log revealed all sorts of random USB bus errors:

usb 6-1.1: USB disconnect, device number 14
usb 6-1: reset full-speed USB device number 13 using uhci_hcd
usb 6-1: device not accepting address 13, error -71
usb 6-1: reset full-speed USB device number 13 using uhci_hcd
usb 6-1: device firmware changed
hub 6-1:1.0: hub_port_status failed (err = -19)
hub 6-1:1.0: hub_port_status failed (err = -19)
hub 6-1:1.0: hub_port_status failed (err = -19)
hub 6-1:1.0: activate --> -19
usb 6-1: USB disconnect, device number 13
usb 6-1: new full-speed USB device number 15 using uhci_hcd
usb 6-1: string descriptor 0 read error: -71
usb 6-1: New USB device found, idVendor=04fe, idProduct=0008
usb 6-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 6-1: can't set config #1, error -71
hub 6-0:1.0: port 1 disabled by hub (EMI?), re-enabling...

This looked like something systemic. Either the controller was resetting continuously or there was something wrong with the USB wiring between the controller and the computer.

A new, identical HHKB model goes for more than $110 today so I opened it up to see if there's anything I can do. After checking the cable with an ohm-meter my suspicion fell on the power supply which seems to consist of a 3.3 V LDO regulator and some capacitors. I could see no obvious transients on the power rails when the controller switched on after the keyboard was plugged in. One interesting thing I did see was that if negotiation with USB host fails the controller switches itself off completely, including its quartz oscillator.

Since I had some problems with flaky USB cables before, I removed the original cable and soldered a new cable directly to the circuit board. This fixed the problem! After poking around some more, it turned out that after re-soldering the connector to the PCB the original cable worked as well.

I removed the membrane before poking around the controller board since a hot soldering iron and plastics don't mix well. Re-inserting the soft matrix tails into the (non-ZIF) connectors was somewhat tricky. I resorted to using pliers plus a bit of paper to protect the delicate wires.

I also noticed that silver wires on the keyboard matrix itself seem to be developing a kind of a dark oxide on the outer edges. I don't remember for sure whether they looked like this from the start though. If something is eating away at the wires that definitely puts a kind of a definitive limit to this keyboard's longevity.

In conclusion, just checking with a multimeter doesn't mean there's not a bad solder joint somewhere on a high-speed bus. The wisdom of checking first for bad RoHS soldering on mechanically (and thermally) stressed components confirmed itself again. Also, did you know there are a couple of alternative, open source Happy Hacking Keyboard controllers out there?

Yesterday I released Unidecode 0.04.14, a new version of my Python port of Sean Burke's Text::Unidecode Perl module for transliterating Unicode strings to 7-bit ASCII.

Together with a few other minor changes, this release reverts one quite controversial change in the previous version. In the new version, Latin characters with diaeresis (ä, ö, ü) are again simply stripped of accents instead of using German transliterations (ae, oe, ue).

Using German transliterations instead of simple language-neutral accent stripping was the most often requested change to Unidecode. However, after that change was released, the most often reported bug was again concerning the transliteration of these characters. This reaction was interesting and several lessons have been learned from it.

Before diving into the problem of transliteration at all, it was obvious that people wrote code under the assumption that Unidecode will not change the transliterations over time. Apparently the 0.04.13 release broke many websites because of this, since the most popular use of Unidecode is to generate ASCII-only URLs from article titles. This is interesting because each and every release of Python Unidecode contained changes to the mappings between Unicode and ASCII characters, as it was clearly stated in the ChangeLog. This bug only became apparent once transliteration of an often used character, like ä, was changed.

So, the lesson here is that if you are using Unidecode for generating article slugs, you should only do transliteration once and store it in the database.

Now, the issue with automatic transliteration is that it's hard to do it right. In fact, it's something that requires understanding of natural language, itself a strong AI problem. If you want to get a rough idea what it involves and what kind of trade-offs Unidecode does, I suggest reading Sean Burke's article about it. Here's a relevant quote from it:

The grand lesson here is tht if y lv lttrs ot, ppl cn stll make sense of it, but ifa yaou gao araounada inasaeratainaga laetataerasa, the result is pretty confusing.

Above explains nicely why having German transliterations in Unidecode doesn't work. While it might be that German is the most common use of these characters, it makes transliteration for other languages using the same characters much worse. As Sean points out, it is much easier for a German speaker to recognize words with missing es than it is for instance for a Finnish reader to deal with extra es (compare Hyvää päivää with Hyvaeae paeivaeae or Wörterbuch with Worterbuch). It was my error that I did not remember this argument before accepting the German transliteration patch.

However, this is such a popular issue that even Wordpress implements the German transliteration as the only language-specific exception in their own transliteration tables. It should also be pointed out though that this simple fix does not actually make German transliteration perfect. There is an issue with capitalization (without understanding the context you can't know whether to replace upper-case Ä with Ae or AE).

The solution here, which I will suggest in the future to anyone that will report this as a bug, is that you should use a language-specific transliteration step before using Unidecode. You can find several of them on the web. Some, like Unihandecode, have been built on top of Unidecode.

One thing you should be aware though, is that these language-specific solutions can give a false sense of correctness. You are now relying on responsible people setting the string language correctly, when many don't even get the string encodings right. Also, can you be sure that all of the input will be in the language that was specified? An occasional foreign visitor might be much more upset with a wrong language-specific transliteration of her name than a somewhat language-neutral one provided by Unidecode.

In any case, you should be aware that using automatic transliteration to produce strings that are visible to general public will lead to such problems. This is something that developers of Plone and Launchpad got to experience first hand (although I believe the latter was not due to Unidecode).

In conclusion, I will now be much more careful accepting patches to Unidecode that deal with language-specific characters. In contrast to Sean I don't have a Master's degree in linguistics and only have a working knowledge of three languages. This makes me mostly unqualified to judge whether proposed changes make sense. Even if submitters put forth good arguments, they probably don't have a complete picture of which other languages their change might affect. Even though they didn't raise as much dust as this most recent one, I'm now actually afraid of how much damage was caused by those other few changes to Sean's original transliteration tables I accepted.

Ever since Debian Wheezy has been released, I've been putting off the upgrade of my non-server machines because I don't want to switch from my cozy GNOME 2 desktop. As kind of a test trial, I've been using Wheezy with XFCE on my work laptop since mid-July. With some work I managed to get it pretty close to GNOME 2 functionality I am used to. However it still feels like a significant step backwards in usability and it has several annoying features that I was so far unable to work around.

I might post a list of tweaks I did later on in another post. For now, here is a somewhat ugly hack to work around the fact that XFCE never fails to do exactly the wrong thing when a monitor has been added or removed from the system. As I often move between a docking station, external monitor and an occasional projector, it's extremely annoying to have to drop into a terminal all the time to poke around xrandr.

This is based on this post I found. The original solution didn't work for me and was a bit more convoluted than it strictly needed to be. Plus having some error checking is always nice.

Any way, put the following into /etc/udev/rules.d/20-monitor.rules to tell udev to call a shell script on a monitor hot plug event:

ACTION=="change", SUBSYSTEM=="drm", ENV{HOTPLUG}=="1", ENV{DEVNAME}=="dri/card0", RUN+="/etc/udev/monitors.sh"

The corresponding script in my case looks like this:

#!/bin/bash

set -e

XUSER=`w -h -s|awk '$3~"^:0"{print $1; exit 0}'`
if [ -z "$XUSER" ]; then
	echo "No one logged in, exiting."
	exit 0;
fi

XHOMEDIR=`getent passwd $XUSER|cut -d: -f6`

export DISPLAY=":0"
export XAUTHORITY="$XHOMEDIR/.Xauthority"

# Adjust the following to fit your needs

function dock {
	xrandr --output DisplayPort-2 --left-of LVDS --primary --auto
}

function undock {
	xrandr --auto
}

if xrandr|grep -q 'DisplayPort-2 connected'; then
	dock
else
	undock
fi

By the way, to debug scripts called from udev, use:

# udevadm control --log-priority=debug

This causes udev to log standard output and error streams emitted by scripts to syslog and is pretty useful when a script mysteriously doesn't seem to do what is supposed to.

The original warnings still apply. This solution only works when there is one graphical log in. It also doesn't work when you first turn on the computer or when the configuration of monitors change while the laptop was suspended.

For these cases I have a shortcut to monitors.sh on my desktop which I can click in anger when my high-DPI 24" flat panel starts pretending it has resolution of a Game Boy LCD.

Here's a recipe how to solve a particular problem with GNU Radio. More or less for my own record as I'm pretty sure I'll come across this same problem again on some other machine and it took some time to figure it out.

If suddenly GNU Radio instrumentation using the WX GUI (like FFT and waterfall plots) look all strange and broken like this:

the cause of the problem is that Python OpenGL bindings are not working for some reason. In this case GNU Radio will automatically fall back onto somewhat simpler widgets without any warnings in the console. These backup widgets however seem to be very outdated or downright broken. A lot of buttons and controls are missing and the waterfall plot is completely useless.

Package that needs to be installed on Debian is python-opengl.

After this is corrected, the same WX GUI FFT Sink as above will be shown like this:

You can override auto-detection of a working OpenGL installation by modifying the following part of etc/gnuradio/conf.d/gr-wxgui.conf. This is useful because if you force the style to gl you will actually see the error that is causing the code to fall back to non-GL widgets:

[wxgui]
# 'gl', 'nongl', or 'auto'
style = auto