Plotting a density map

28.11.2011 20:04

Density maps are one of those things I need rarely enough that I forget how I did it last time but also often enough that spending 15 minutes searching on Google and browsing Octave manual seems unnecessary.

So, for my reference and yours, here's the simplest, fastest way I know of producing one on a vanilla GNU/Linux box. Density plot is something Gnuplot doesn't know how to do by itself. So you need to fire up Octave:

octave> [c,x,y] = hist2d(load("foo"), 150, 150); imagesc(x,y,c)

Here, foo is a tab-separated-values text file with two columns: X and Y coordinate of each point. The two numerical arguments to hist2d are the numbers of buckets for each axis. This produces a plot like this by default:

Example density map

As with all Octave plots, you can write the current picture to a file using:

octave> print -dpng foo.png

On Debian you need octave-plot package installed for hist2d to work. See Octave manual for ways to further customize the look.

By the way, you might have noticed that the Y axis is up-side-down on the example plot above. That seems to be a bug in Octave when you only have negative values on the axis and I haven't yet figured out how to work around it.

Update: Running set(gca,'ydir','normal'); after the imagesc command restores the usual orientation of the axes for image plots.

Posted by Tomaž | Categories: Code | Comments »

ARM primer

24.11.2011 15:10

My first somewhat serious look into the ARM architecture was at the CCC Camp, where I attempted to extract the secret signing keys via a buffer overflow exploit in the r0ket. Recently I started working on a bit more serious project that also involves copious amount of C code compiled for ARM and I thought I might give a quick overview of the little important details I learned in the last few weeks.

STM32F103 ARM microcontroller

First of all, targeting ARM is tricky business. As the Wikipedia article nicely explains, you have multiple families (versions) of ARM cores. They are slightly incompatible, since instructions and features have been added and deprecated over time. Recent cores are divided between application, real-time and microcontroller profiles. You find application cores (say Cortex-A9) in iPads, phones and other gadgets that need a fancy CPU with cache, SMP and so on, while microcontrollers obviously omit most of that. The most popular microcontroller profile core right now seems to be Cortex-M3 - where Cortex is a code name for ARM version 7 and M3 stands for the third microcontroller profile. However, this is still a very broad term, since these cores get licensed by different vendors that put all kinds of their own proprietary periphery (RAM, non-volatile storage, timers, ...) around them. So stating the fact that you target Cortex-M3 is much less specific than, say, targeting x86 where you have de-facto standards for at least some basic components around the CPU. That's why the current Linux kernel lists 64 ARM machine types compared with practically just one for x86.

Speaking of Intel's x86, people like to say how it is full of weird legacy quirks, ARM has some of its own. One thing that caught me completely by surprise is the Thumb instruction set. It turns out the that original dream of a clean, simple design with a fixed instruction length didn't play out that well. So ARM today supports two different instruction sets, the old fixed-length one and a new variable-length "Thumb" set that mostly has a one-to-one relation to the old one. In fact, the CPU is capable of switching between instruction sets on function call boundary, so you can even mix both in a single binary.

To add to the confusion, there are two function call conventions, meaning you have to be careful when mixing binaries from different compilers. There's also a mess regarding hardware floating point instructions, but thankfully not many microcontrollers include those. Debian Wiki nicely documents some of the finer details concerning ARM binary compatibility.

All of these issues mean there's a similar state of chaos regarding cross-compiler toolchains. After what seems like many discussions Debian developers haven't yet come to a consensus on what kind flavor to include in the distribution. That means you can't simply apt-get install arm-gcc, like you can for AVR. Embedian provides some cross-compiler binaries, but those are targeted at compiling binaries for ARM port of Linux, not bare-bones systems for microcontrollers. You can build stripped-down binaries with such a compiler, but expect trouble since build systems for microcontroller software don't expect that. From what I've seen the most popular option by far appears to be CodeSourcery, a binary distribution of the GNU compiler toolchain. The large binary blob installer for Linux made my hair stand up. Luckily there's an alternative called summon-arm-toolchain which pieces together a rough equivalent after a couple of hours of downloading and compiling. If this particular one doesn't do the trick for you, there's a ton of slightly tweaked GitHub forks out there.

By the way, GCC has a weird naming policy for ARM-related targets. For instance, CodeSourcery contains arm-elf-gcc and arm-none-eabi-gcc (where arm-elf and arm-none-eabi refer to the --target configure option when compiling GCC). Both will in fact produce ELF object files and I'm not sure what the actual difference is. Some say that it's the ABI (function call convention) and in fact you can't link binaries from one with binaries from the other. However you can control both the ABI and the instruction set at run-time through the GCC's target machine options. Other than that, they seem to be freely interchangeable.

For uploading the software to the microcontroller's flash ROM I'm using the Olimex ARM-USB-OCD with the openocd, which also seems to be a popular option. It allows you to program the microcontroller through JTAG and is capable of interfacing with the GNU debugger. This means you can upload code, inspect memory and debug with single-stepping and break points from GDB's command line in much the same way you would a local process. There are some quirks to get it running though, namely OpenOCD 0.5.0 doesn't like recent versions of GDB (a version that works for sure is 7.0.1).

I think this kind of wraps it up. As always, a reference manual of the particular processor you use at the reach of a hand is a must. When things stubbornly refuse to work correctly it's good to fall back to a disassembled listing of your program (objdump -d) to check for any obvious linking snafus. And, when trust in all other parts of your toolchain fails and sharks a circling ever closer, a trusty LED is still sometimes an indispensable debugging equipment.

Posted by Tomaž | Categories: Code | Comments »

It even has a fingerprint

15.11.2011 18:18

I was going low on my personal business cards, so I ordered a new batch from Moo. Besides adding some new circuit-porn designs for the back side I also added the fingerprint of my GPG key next to my e-mail address on the front. When I flashed a bunch of cards in Kiberpipa the other day, Brodul pointed me to this policy for business cards of KDE developers that says that GPG fingerprints on the cards give an unprofessional impression.

My new batch of Moo cards

Interesting. After a bit of browsing the opinion on this matter in the open source world seems to be divided. I remember that cards from Red Hat were the first that I saw with the fingerprint displayed. However there's also this interesting complaint that it has only ever been used to mock the owner of the card. They also seem to be popular with Debian developers and people were seen complaining that it has been dropped from Ubuntu's design.

I actually thought for a while whether to add it or not before submitting the design. For instance, it makes the little Moo card much more cluttered and I liked the previous simplicity of three lines with an e-mail and web address. On the other hand I like to promote secure communication and have been consistently cryptographically signing my mail for many years now. So you can also take it as a statement I guess.

Part of why I think sharing your public key in this way is a good idea is because in my opinion trusting keys based on traditional key-signing parties was a big blunder. There you had 10 random people, signing keys based on some government-issued identification. What this does is only to transfer your trust in government ID to trust in the key. Let's not even go to the problem of me not knowing what the official IDs look like in most countries or the fact that I don't necessarily trust your government. You could score a hundred signatures like that and that doesn't help me in knowing whether the key owner is the person behind the face I met here-and-here.

So I am now willing to sign a public key after I spent some time with the person that personally gave me that key. Enough time to convince me that he is who he claims to be. And I encourage other people to do the same when signing my key (a ritual, by the way, a certain social scientist once described as fascinating).

Yes, this method is vulnerable to a social engineering attack. But so is signing a key based on someone's driver's license and I am sure social interaction is harder to fake than a foreign-looking personal identification card. By getting a fingerprinted business card you get a hard-to-undetectably-change hard-copy of the fingerprint, for which you can be reasonably sure that it belongs to the person that gave it to you.

Posted by Tomaž | Categories: Life | Comments »

News from the .7 m band

06.11.2011 18:06

Here's a quick update on my multipurpose receiver project for the 433 MHz ISM band.

I've managed to get the noise problem somewhat under control. The interference from the USB module is mostly gone now that I have a RF shield around that part. However it still takes some practice to get a clear signal from the receiver module. As it is, I still can't use the full length λ/4 antenna or all I get from the receiver is noise. So I usually reduce the sensitivity by using somewhat shorter lengths of wire. Finding the right location for the receiver also seems to be important, which suggests that the noise I receive comes from localized sources and isn't inherent to the receiver or the ambient. Anyway, I now mostly use the receiver with a 1 m USB extension cable that allows me to find the quietest spot on the table.

433 MHz receiver connected to an EeePC

From the software side I now have a pretty useful system that consists of several layers:

  • Stand-alone process that reads a baseband stream from hardware, automatically detects several kinds of digital modulations (PWM, FSK, manchester and some weird ones I saw in the wild) and outputs a decoded binary packet plus some physical-level header that includes the modulation data, timestamp and so on.
  • Patched libpcap that reads packets from this process and provides a standard packet-capture interface on top of it.
  • Patched tcpdump with dissectors for packets produced by popular encoder ICs (such as MC145026 and Superstar TX-13) and some heuristics for determining the actual meaning behind the encoded bits.

The final result then looks somewhat like this:

$ tcpdump -X -i am433_1_0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on am433_1_0, link-type 254, capture size 65535 bytes
02:42:56.101249 technoline weather dev 15 chan 3 [22.3 C inc 60 % RH] 0 1
        0x0000:  bf2f b01b 80                             ./...

Meaning that there's a weather monitor somewhere in range that reports a balmy 22.3°C with 60% relative humidity.

All this actually turned out to be a quite useful diagnostic tool. So far I used it to repair Kiberpipa's door remote and take some measurements with my own remote weather sensor. And just running it at different locations and observing various packets flying around is fun on its own.

Actually, my first idea was to use Wireshark instead of tcpdump, because I wanted to write dissectors in one of the high-level languages it supports and because its GUI offers some quite convenient filtering functions. However that beast turned out to be somewhat complicated and I didn't found the patience to deal with it. Actually, libpcap and tcpdump themselves were more messy than I expected, but I guess supporting all those combinations of OS and networks comes with a cost.

I'll be publishing most of this work under an open license shortly in some public git repository. I would have done that sooner, but from the start my private repository has accumulated a large number of packet captures for testing purposes and those might include packets that I can't make public. For those interested in knowing more I will also be giving a talk on all of this in Kiberpipa in December.

Posted by Tomaž | Categories: Digital | Comments »